Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XXX...XX.X.] duration: 1.086269105s FAIL (2025-08-07 23:22:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.086300263s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 319.032673ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:22:47 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 319.072345ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.................................................X.XXXXX.X..] duration: 12.325176266s FAIL (2025-08-07 23:22:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.32522162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [............................................XXXXX.XX.XX.X...] duration: 896.307559ms FAIL (2025-08-07 23:22:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 896.337877ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X.XX..X......] duration: 3.366725394s FAIL (2025-08-07 23:22:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.36675019s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X....X............X......................] duration: 3.153389606s FAIL (2025-08-07 23:22:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.153425468s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................................................XXXXXXXXXX..] duration: 3.101761371s FAIL (2025-08-07 23:22:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 3.101797178s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..................................................X.XXXXX.X.] duration: 22.61103559s FAIL (2025-08-07 23:22:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.611069443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X...X.XX...] duration: 3.124850195s FAIL (2025-08-07 23:22:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.124874907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.279405594s FAIL (2025-08-07 23:22:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.279437714s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X...X.XX..] duration: 19.911196547s FAIL (2025-08-07 23:22:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 19.911312954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XXX...XX.X] duration: 30.000779405s FAIL (2025-08-07 23:22:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000803938s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X....X............X.....................] duration: 3.211378433s FAIL (2025-08-07 23:22:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.211414114s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 621.511636ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:22:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 621.539159ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................................................XXXXXX.XX...] duration: 2.534498828s FAIL (2025-08-07 23:22:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.534570581s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.............................................XXX.XXXXXX..XX.] duration: 1.146906263s FAIL (2025-08-07 23:22:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Aug 7 23:21:07.375939272 (13h41m29.119996122s) [upload=14096]: Starting upload Aug 7 23:21:07.376022371 (13h41m29.120079233s) [upload=14096]: Done starting upload Aug 7 23:21:07.376131644 (13h41m29.120188494s) [upload=14096]: Uploading segments... Aug 7 23:21:07.376228707 (13h41m29.120285578s) [upload=14096]: Got next segment. Inline: true Aug 7 23:21:07.376236415 (13h41m29.120293285s) [upload=14096]: Waiting for error group managing segments... Aug 7 23:21:07.376314696 (13h41m29.120371567s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:07.376322308 (13h41m29.120379199s) [upload=14096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:21:07.376327371 (13h41m29.120384242s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:37.104686329 (13h41m58.84874319s) [upload=14096]: Done uploading segments... Aug 7 23:21:37.104719076 (13h41m58.848775939s) [upload=14096]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.146946071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [.............................................XXXXX.XX.XX.X..] duration: 856.671819ms FAIL (2025-08-07 23:21:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 856.708194ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X....X............X....................] duration: 2.725281581s FAIL (2025-08-07 23:21:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 2.725310071s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXX.] duration: 2.558630665s FAIL (2025-08-07 23:21:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 2.558674765s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...................................................X.XXXXX.X] duration: 1m0.001043603s FAIL (2025-08-07 23:21:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001061954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [..............................................XXXXX.XX.XX.X.] duration: 934.394596ms FAIL (2025-08-07 23:21:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 934.42724ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 11 out of 60 is failed (threshold=3) [..............................................XXX.XXXXXX..XX] duration: 30.001679327s FAIL (2025-08-07 23:21:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Aug 7 23:21:07.375939272 (13h41m29.119996122s) [upload=14096]: Starting upload Aug 7 23:21:07.376022371 (13h41m29.120079233s) [upload=14096]: Done starting upload Aug 7 23:21:07.376131644 (13h41m29.120188494s) [upload=14096]: Uploading segments... Aug 7 23:21:07.376228707 (13h41m29.120285578s) [upload=14096]: Got next segment. Inline: true Aug 7 23:21:07.376236415 (13h41m29.120293285s) [upload=14096]: Waiting for error group managing segments... Aug 7 23:21:07.376314696 (13h41m29.120371567s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:07.376322308 (13h41m29.120379199s) [upload=14096]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:21:07.376327371 (13h41m29.120384242s) [upload=14096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:37.104686329 (13h41m58.84874319s) [upload=14096]: Done uploading segments... Aug 7 23:21:37.104719076 (13h41m58.848775939s) [upload=14096]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001727807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [................................................XX..XXXXX...] duration: 1.189341062s FAIL (2025-08-07 23:21:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.189378723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [...............................................XXXXX.XX.XX.X] duration: 30.00071016s FAIL (2025-08-07 23:21:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000739059s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X.XX..X.....] duration: 4.121335801s FAIL (2025-08-07 23:21:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.121366985s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................................XXXXXX.XX..] duration: 30.003490455s FAIL (2025-08-07 23:21:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.003520222s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X...X.XX.] duration: 9.736373442s FAIL (2025-08-07 23:21:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.736418502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.115000303s FAIL (2025-08-07 23:21:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.115035109s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X.XX..X....] duration: 9.932305814s FAIL (2025-08-07 23:21:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.932341194s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 311.325407ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:21:15 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 311.359406ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X....X............X...................] duration: 3.1433932s FAIL (2025-08-07 23:21:09 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.143441636s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X...X.XX] duration: 1m0.290182646s FAIL (2025-08-07 23:21:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== inished. err: context deadline exceeded Aug 7 23:20:07.202369543 (13h40m28.946426394s) [upload=14082]: Starting upload Aug 7 23:20:07.202426657 (13h40m28.946483509s) [upload=14082]: Done starting upload Aug 7 23:20:07.202696950 (13h40m28.946753814s) [upload=14082]: Uploading segments... Aug 7 23:20:07.202761256 (13h40m28.94681811s) [upload=14082]: Got next segment. Inline: false Aug 7 23:20:07.202807046 (13h40m28.946863908s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:07.202814233 (13h40m28.946871102s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:45.722093118 (13h41m7.466149983s) [seg_pos={0 0}, upload=14082]: Begin upload segment... Aug 7 23:20:45.722177165 (13h41m7.466234018s) [seg_pos={0 0}, upload=14082]: Done begin upload segment. Aug 7 23:20:45.722218538 (13h41m7.4662754s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722246298 (13h41m7.46630316s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722330454 (13h41m7.466387316s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722658583 (13h41m7.466715506s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722717191 (13h41m7.46677405s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.722989701 (13h41m7.467046568s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723061064 (13h41m7.467117927s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.723458393 (13h41m7.467515259s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.729867582 (13h41m7.473924444s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.731063643 (13h41m7.475120512s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.732350031 (13h41m7.476406894s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.733989697 (13h41m7.478046556s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734727605 (13h41m7.478784474s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.734786637 (13h41m7.478843504s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735460668 (13h41m7.479517532s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.735486537 (13h41m7.479543401s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.736565943 (13h41m7.480622804s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:45.737386328 (13h41m7.48144319s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Uploading piece... Aug 7 23:20:46.299198843 (13h41m8.043255689s) [upload=14082]: Next returned nil segment Aug 7 23:20:46.299217870 (13h41m8.043274733s) [upload=14082]: Waiting for error group managing segments... Aug 7 23:20:46.464881860 (13h41m8.208938721s) [noise=true , address=104.36.82.60:4345 , piece_id=2AKG3BCGW7TFNVG3FQ576YJOTEZV3V2NP5ITSSZ7BO3NO5CRVDVQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.473775294 (13h41m8.217832164s) [noise=true , address=104.36.82.52:4106 , piece_id=KTACAAM5SP3PCN3VE6CKF3KHMLGYCHONLC7FDUI4YHLFKIEJ65HQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.474664669 (13h41m8.218721541s) [noise=true , address=104.36.82.9:4016 , piece_id=VTTRIIBZ4KCGJW2H5DMG7J63AGFIV7QT34MJJTHRUGFIH6JD6EQQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.479688336 (13h41m8.223745197s) [noise=true , address=104.36.82.36:4651 , piece_id=5HQECPYGS5S43JV2CEJY3MRS33NUE6CJ5WPXKTCWYLYVXUQDXLXQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.485171804 (13h41m8.229228658s) [noise=true , address=104.36.82.6:4901 , piece_id=7VX47BXB37LW2CKYFTBTFAJSDF3QLHEHQHRD5ZM5EHZPAZFPXBPQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.486836180 (13h41m8.230893032s) [noise=true , address=104.36.82.61:4392 , piece_id=EL44CRMZDDTXOW45GURIDIFQF4K65UUJFKR6L3F3WQBMVXSD2XHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.496476947 (13h41m8.240533796s) [noise=true , address=104.36.82.44:4876 , piece_id=3WUHXX7GVGZWOGDA7FETGVT772RLHO4HRV6OCZKNTZIMG3VDPDOA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.556494206 (13h41m8.300551081s) [noise=true , address=104.36.82.16:4323 , piece_id=RQYKNED6IWTV47W7HGZCVVOUKFRMHGO5WSPCW36ROCXMEDEFYM4Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.628743021 (13h41m8.372799882s) [noise=true , address=45.91.116.14:4201 , piece_id=LCOEAQDFX432OULFXCKDH3FQMZMPR5XSDVKK5MAPEQYEABUAYMDA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.645987246 (13h41m8.390044106s) [noise=true , address=104.36.82.4:4813 , piece_id=IZO235F25YVNCQG2GSR3UPI5OPMONHVRQQAWDEBOZWR2EILWGCCA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.666511132 (13h41m8.410567992s) [noise=true , address=104.36.82.5:4867 , piece_id=IEDY33GIP4MZRSN2MKKBLDFAXPMKJUPZVPMOXVCEFJ6BUPZJHYHQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.671756371 (13h41m8.41581322s) [noise=true , address=104.36.82.37:4680 , piece_id=2MDE2TCPX2ROXJXK7Q234YXMK4BFHPDQB5EB7GECHLMUJY5W4ITQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.934025878 (13h41m8.678082729s) [noise=true , address=79.127.205.234:4597 , piece_id=WETMAYMNBWD6F6JYCXPNAG5KOJXNCXE3KM7ZJ4F5Y3GEBPHLMDEA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970907150 (13h41m8.714964016s) [noise=true , address=64.49.13.139:4474 , piece_id=LGXAPMGRQOOEXXG7EBNLO7GAPDKFBCQIBWGYID7KOVQ3UNMDZICQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: <nil> Aug 7 23:20:46.970940415 (13h41m8.714997268s) [seg_pos={0 0}, upload=14082]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:46.971273207 (13h41m8.715330074s) [noise=true , address=64.49.13.155:4916 , piece_id=PJ3VBT33DUHLG3WHE6AK7IIYX76IZZFJCAWX7WX3UUBA6BLGB5VA, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971460895 (13h41m8.715517755s) [noise=true , address=64.49.13.169:4705 , piece_id=YJCLDFSO5ENFREIROYPEZQQTQD4GM4A3DFKNJEHSPFDOKDCFEJ7Q, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971541847 (13h41m8.715598706s) [noise=true , address=79.127.205.246:4276 , piece_id=6753PO7TQBULISFOSTSIT32I7DSKQPM6M4TEVF2E7TILSMUU5ARQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJTNVtPFN48S9ZrayehLD2HcVckyJWLx2rtEvyH56W22R8Suq): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971599101 (13h41m8.715655962s) [noise=true , address=104.36.82.58:4292 , piece_id=KQ2RNSZDMCU7OED4IJC4ERBTMTYVGHJCRSVOFZUXG24LW7H3KHUQ, seg_pos={0 0}, upload=14082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:46.971618718 (13h41m8.715675568s) [seg_pos={0 0}, upload=14082]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:46.971698846 (13h41m8.715755698s) [upload=14082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:46.971704848 (13h41m8.71576171s) [upload=14082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:21:07.219673652 (13h41m28.963730505s) [upload=14082]: Done uploading segments... Aug 7 23:21:07.219701493 (13h41m28.963758344s) [upload=14082]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.290204373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.0010163s FAIL (2025-08-07 23:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:20:34.909564598 (13h40m56.65362148s) [upload=14088]: Starting upload Aug 7 23:20:34.909657672 (13h40m56.653714552s) [upload=14088]: Done starting upload Aug 7 23:20:34.909836458 (13h40m56.653893325s) [upload=14088]: Uploading segments... Aug 7 23:20:34.910062213 (13h40m56.654119064s) [upload=14088]: Got next segment. Inline: false Aug 7 23:20:34.910103244 (13h40m56.654160095s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:34.910108545 (13h40m56.654165405s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:21:04.641072069 (13h41m26.385128931s) [upload=14088]: Next segment err: context deadline exceeded Aug 7 23:21:04.641101566 (13h41m26.385158427s) [upload=14088]: Done uploading segments... Aug 7 23:21:04.641122347 (13h41m26.3851792s) [upload=14088]: Upload finished. err: context deadline exceeded Duration: 30.001047146s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXX...XX.] duration: 1.096707387s FAIL (2025-08-07 23:21:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.096756168s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....X............X..................] duration: 2.88105701s FAIL (2025-08-07 23:20:50 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 2.881099769s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 302.754372ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:20:50 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 302.78677ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 10 out of 60 is failed (threshold=3) [...............................................XXX.XXXXXX..X] duration: 30.001657825s FAIL (2025-08-07 23:20:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:13:33.400642971 (13h33m55.144699817s) [upload=13970]: Done starting upload Aug 7 23:13:33.400723341 (13h33m55.144780187s) [upload=13970]: Uploading segments... Aug 7 23:13:33.400775896 (13h33m55.144832764s) [upload=13970]: Got next segment. Inline: true Aug 7 23:13:33.400780746 (13h33m55.144837592s) [upload=13970]: Waiting for error group managing segments... Aug 7 23:13:33.400844129 (13h33m55.144900979s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:33.400851223 (13h33m55.144908092s) [upload=13970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:33.400854888 (13h33m55.144911755s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:33.576404182 (13h33m55.320461054s) [upload=13970]: Done uploading segments... Aug 7 23:13:33.576428692 (13h33m55.320485559s) [upload=13970]: Upload finished. err: <nil> Aug 7 23:13:55.968242604 (13h34m17.712299464s) [upload=13977]: Starting upload Aug 7 23:13:55.968304238 (13h34m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:52.160899256 (13h39m13.904956117s) [upload=14060]: Starting upload Aug 7 23:18:52.160970297 (13h39m13.905027157s) [upload=14060]: Done starting upload Aug 7 23:18:52.161065587 (13h39m13.905122439s) [upload=14060]: Uploading segments... Aug 7 23:18:52.161108065 (13h39m13.905164934s) [upload=14060]: Got next segment. Inline: true Aug 7 23:18:52.161113994 (13h39m13.905170844s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:18:52.161171289 (13h39m13.90522814s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:52.161176753 (13h39m13.905233614s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:52.161180292 (13h39m13.905237155s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:52.299645097 (13h39m14.043701961s) [upload=14060]: Done uploading segments... Aug 7 23:18:52.299672156 (13h39m14.043729007s) [upload=14060]: Upload finished. err: <nil> Aug 7 23:19:29.899969190 (13h39m51.64402605s) [upload=14071]: Starting upload Aug 7 23:19:29.900023963 (13h39m51.644080812s) [upload=14071]: Done starting upload Aug 7 23:19:29.900097059 (13h39m51.644153908s) [upload=14071]: Uploading segments... Aug 7 23:19:29.900152738 (13h39m51.644209608s) [upload=14071]: Got next segment. Inline: true Aug 7 23:19:29.900161132 (13h39m51.644218002s) [upload=14071]: Waiting for error group managing segments... Aug 7 23:19:29.900212688 (13h39m51.64426955s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:29.900219311 (13h39m51.644276178s) [upload=14071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:29.900224731 (13h39m51.64428173s) [upload=14071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:30.208671256 (13h39m51.952728106s) [upload=14071]: Done uploading segments... Aug 7 23:19:30.208715246 (13h39m51.952772096s) [upload=14071]: Upload finished. err: <nil> Aug 7 23:20:17.640185376 (13h40m39.384242236s) [upload=14084]: Starting upload Aug 7 23:20:17.640240796 (13h40m39.384297642s) [upload=14084]: Done starting upload Aug 7 23:20:17.640310076 (13h40m39.384366923s) [upload=14084]: Uploading segments... Aug 7 23:20:17.640362556 (13h40m39.384419425s) [upload=14084]: Got next segment. Inline: true Aug 7 23:20:17.640367233 (13h40m39.384424093s) [upload=14084]: Waiting for error group managing segments... Aug 7 23:20:17.640417842 (13h40m39.384474688s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:20:17.640422768 (13h40m39.384479628s) [upload=14084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:20:17.640425589 (13h40m39.384482436s) [upload=14084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:21.993094963 (13h40m43.737151826s) [upload=14084]: Done uploading segments... Aug 7 23:20:21.993114705 (13h40m43.737171555s) [upload=14084]: Upload finished. err: <nil> Duration: 30.001683079s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....................................................X.XXXXX.] duration: 1m0.001335551s FAIL (2025-08-07 23:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.0013497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 2m9.657468098s FAIL (2025-08-07 23:20:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load Aug 7 23:17:49.307705424 (13h38m11.051762272s) [upload=14039]: Done starting upload Aug 7 23:17:49.308016567 (13h38m11.052073415s) [upload=14039]: Uploading segments... Aug 7 23:17:49.308064138 (13h38m11.052120985s) [upload=14039]: Got next segment. Inline: false Aug 7 23:17:49.308123883 (13h38m11.052180743s) [upload=14039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:49.308129375 (13h38m11.052186235s) [upload=14039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:19.032758502 (13h38m40.776815367s) [upload=14039]: Next segment err: context deadline exceeded Aug 7 23:18:19.032971931 (13h38m40.777028782s) [upload=14039]: Done uploading segments... Aug 7 23:18:19.033014318 (13h38m40.777071179s) [upload=14039]: Upload finished. err: context deadline exceeded Aug 7 23:18:25.127072226 (13h38m46.871129088s) [upload=14052]: Starting upload Aug 7 23:18:25.127322319 (13h38m46.871379183s) [upload=14052]: Done starting upload Aug 7 23:18:25.127477364 (13h38m46.871534223s) [upload=14052]: Uploading segments... Aug 7 23:18:25.127943505 (13h38m46.872000369s) [upload=14052]: Got next segment. Inline: false Aug 7 23:18:25.128022972 (13h38m46.87207984s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:25.128032328 (13h38m46.872089195s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:44.923639390 (13h39m6.667696251s) [seg_pos={0 0}, upload=14052]: Begin upload segment... Aug 7 23:18:44.923694548 (13h39m6.667751411s) [seg_pos={0 0}, upload=14052]: Done begin upload segment. Aug 7 23:18:44.923719227 (13h39m6.667776093s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.923742940 (13h39m6.667799806s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924062820 (13h39m6.66811968s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924231371 (13h39m6.668288235s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924310412 (13h39m6.668367275s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924552072 (13h39m6.668608933s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.924795545 (13h39m6.66885246s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925037101 (13h39m6.669093961s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925293050 (13h39m6.669349911s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925526381 (13h39m6.669583247s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.925876037 (13h39m6.669932897s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926278285 (13h39m6.670335152s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926413382 (13h39m6.670470243s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.926732421 (13h39m6.670789284s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927153074 (13h39m6.671209983s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:44.927375228 (13h39m6.671432108s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Uploading piece... Aug 7 23:18:45.586845311 (13h39m7.330902181s) [upload=14052]: Next returned nil segment Aug 7 23:18:45.586873064 (13h39m7.330929934s) [upload=14052]: Waiting for error group managing segments... Aug 7 23:18:45.727354327 (13h39m7.471411192s) [noise=true , address=104.36.82.10:4079 , piece_id=HY5F64KPLN2T7YZH3YYSPSG3ZPMGBLWNI4I3PKKZYNZ5YEZR4GVA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.767876801 (13h39m7.511933648s) [noise=true , address=104.36.82.6:4873 , piece_id=3SJJMGDMIWR5EFT6MKMBSVOHDT4FLRGF4IJLY5M7TEBLA4BDFBEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.792222311 (13h39m7.536279182s) [noise=true , address=104.36.82.7:4912 , piece_id=G2UDIUZAL6D4KWUVSVG723QHFXKPENHBZCOW6GU6MJERQ5CLMRFA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.836090391 (13h39m7.580147242s) [noise=true , address=45.91.116.13:4154 , piece_id=DQLQNTJEGGOFBOB7BYQK7PI6TLW62QXR7NZ3A7FSRNJTOZZOAQWQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.847910967 (13h39m7.591967821s) [noise=true , address=45.91.116.6:4973 , piece_id=SAKE474GCCI65TYWECTKW56TG64OWZFPQ4LCCYWEDPAEMTFZQRCQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.874060717 (13h39m7.618117569s) [noise=true , address=77.90.31.13:4628 , piece_id=M4D7E36XVNTFZ33KR2CCRH3YSUCRQ7SY4SFWSJC6742OMZ4OMP5Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.882796683 (13h39m7.626853529s) [noise=true , address=77.90.31.12:4589 , piece_id=7GPB4HNYKYBBBM4WVJNOB3DMOS4CGUHPMXMYD6XBXID7SQWINDAQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.883366192 (13h39m7.62742305s) [noise=true , address=45.91.116.12:4226 , piece_id=6W56O7SILRUME7IECKW6JAXORM4LTV74AJYFCGG3MUXB6SAMFHSQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.899253981 (13h39m7.643310835s) [noise=true , address=77.90.31.17:4770 , piece_id=TS5LQVVG4GURERET6BWDTVCSBMFJQR7SLTH276RQMGDXDRFKYVKA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.905112812 (13h39m7.64916966s) [noise=true , address=45.91.116.11:4117 , piece_id=AESPDTGWFCRALUUOFYUJMD7QQZLFKV3YETVW4FFRJ4TQRKG5MF3Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.911636868 (13h39m7.655693715s) [noise=true , address=45.91.116.11:4119 , piece_id=DOIWBHL4FTX74DXSOP3ZBOVHJ4T7C5FXHTSQFABM5BEWWVBXY52Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.934683670 (13h39m7.67874052s) [noise=true , address=104.36.82.11:4115 , piece_id=J24RIZU54XKNVXPGJZA4AZKU7NDHICCMAPOA3GHWAGSOEBSD3AEA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959619489 (13h39m7.703676335s) [noise=true , address=45.91.116.14:4189 , piece_id=QV5LBOR2W73HHMG4YY6RQVVKCVCYYZEKCIST4DSFUITSNXMSTGFQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: <nil> Aug 7 23:18:45.959647576 (13h39m7.703704436s) [seg_pos={0 0}, upload=14052]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:45.959785097 (13h39m7.703841946s) [noise=true , address=45.91.116.8:4045 , piece_id=4QA37GNTHQXLN2SXZN7BX5EJZRIYVHF3K5C4KR2DZ7ZR57V6UTKQ, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959873513 (13h39m7.703930376s) [noise=true , address=45.91.116.4:4939 , piece_id=YWVXZBH3NGWELSESW65VJIGE5YXZMTV7ODEV3J6U3N4W3B3ARUOA, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:45.959997706 (13h39m7.704054568s) [noise=true , address=104.36.82.16:4294 , piece_id=PCHV37SDE6HMSXZUTNSMOTDK3BZZLG4GWAYZVQU6P6GWM6RAUA4Q, seg_pos={0 0}, upload=14052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33694->104.36.82.16:4294: use of closed network connection Aug 7 23:18:45.960040133 (13h39m7.70409698s) [seg_pos={0 0}, upload=14052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:45.960143346 (13h39m7.704200223s) [upload=14052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:45.960151195 (13h39m7.704208074s) [upload=14052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:34.514397794 (13h40m56.258454651s) [upload=14052]: Done uploading segments... Aug 7 23:20:34.514434819 (13h40m56.258491687s) [upload=14052]: Upload finished. err: metaclient: context deadline exceeded Duration: 2m9.657497724s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXX...XX] duration: 30.00044498s FAIL (2025-08-07 23:20:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000629424s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.272650601s FAIL (2025-08-07 23:20:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.272682988s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X.XX..X...] duration: 2.565734835s FAIL (2025-08-07 23:20:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.565783431s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XX..XXXXX..] duration: 896.959798ms FAIL (2025-08-07 23:20:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 897.000056ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.045050883s FAIL (2025-08-07 23:20:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.045147295s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X...X.X] duration: 1m0.000826249s FAIL (2025-08-07 23:20:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== m0.783013536s) [noise=true , address=104.36.82.44:4880 , piece_id=R252YCXF3V3BYKMAMYDQZB62WYCSS2NLI3JTYQPUX5B3NUVMGEMA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.038974664 (13h39m0.783031534s) [noise=true , address=45.91.116.10:4115 , piece_id=5KYVEX26W4V4YIFPU5Q4PCFNFRROYMCIZ6GX7BGXQ5ZS2VKWPZRQ, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.039537724 (13h39m0.783594594s) [noise=true , address=104.36.82.21:4487 , piece_id=MMS5K4JLTMWJLIIYSYLW3ALCLESVEYDFL56TLWDXWKWJKWQ2K7HA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.039551678 (13h39m0.783608532s) [noise=true , address=104.36.82.38:4691 , piece_id=T6ULTGPNZHDN4FIK2QQ4XXPUTXHBYSDG5ZACGYNWGXDSYOJEYXEQ, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.039891351 (13h39m0.783948214s) [noise=true , address=45.91.116.6:4003 , piece_id=J2MJMYJMZKJFTBDT2DTO43YDBJVUSD2A7HHG3N4GUCRA7BSORCQA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.039990416 (13h39m0.784047277s) [noise=true , address=104.36.82.47:4552 , piece_id=XMRBJCTRZHKASCQZOG6JJ235AN3LJLQTZ6SRNXGFHJEYRLSRR76A, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.040237713 (13h39m0.784294584s) [noise=true , address=104.36.82.9:4013 , piece_id=R27TXVFKJWQDJQYGL4GL5LUHO7AIMLICSWY4MUL6XOS7VHLS5KCA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.040441314 (13h39m0.784498194s) [noise=true , address=104.36.82.55:4197 , piece_id=QKOQS7SJ2PKH7NX3JHQX27B67GF62HB2GTFSEYHEBUJ3A526SOBQ, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.040570459 (13h39m0.784627319s) [noise=true , address=79.127.205.233:4625 , piece_id=2IA76NOFLNV7REJEBJ4Y6OUIS7JZVBCLZAA7HYEG3H3DOUAUTZGA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.040846904 (13h39m0.784903755s) [noise=true , address=104.36.82.13:4188 , piece_id=RY6DAZKVGETUITCS2OEQNDKYWHPCLCQ34BADZFGSJLQSE7GJR5SQ, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.041250604 (13h39m0.785307471s) [noise=true , address=104.36.82.62:4423 , piece_id=KF2IMNMXFWBVBXEWWXR46CNOHHWUE7IJA2BU7LLWT37GOWOPQNSQ, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.041679325 (13h39m0.785736186s) [noise=true , address=79.127.205.246:4265 , piece_id=RYBO73UDU2XDRCKURFIHXCXOTZX7SKCS6Y3TXYNA3HAHBQQXAAQQ, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.041696849 (13h39m0.785753704s) [noise=true , address=104.36.82.60:4371 , piece_id=I5D4P4SLDOTDK5OP73BOJMOXYZMVDGOSHYOTEXMN2WF6JH6THSVA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.042073168 (13h39m0.78613002s) [noise=true , address=104.36.82.43:4850 , piece_id=NADQYGWW4ZSUM2H7VZOTGBDF7IHDOHN4XWW7PN22LWVN5X36XNNQ, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.042398007 (13h39m0.786454857s) [noise=true , address=104.36.82.5:4863 , piece_id=JAKGK5M3DUOS5QHRDBSU7CKQ5AK3JPJWTMO4FVD77RXS5UXBI3NA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.042762569 (13h39m0.786819417s) [noise=true , address=156.146.43.225:4449 , piece_id=HUNFO56CIH7SUSQKWDXEKPS5VNOMME7RTFPELRRCJKKCINYMBQYA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.043185884 (13h39m0.787242735s) [noise=true , address=45.91.116.9:4269 , piece_id=R5Z23ICCK5IW24TEJFTX4ZV2SFTB2JJOGPWPMJM3KU4TC7F4GO5A, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.043524179 (13h39m0.787581039s) [noise=true , address=79.127.201.215:4357 , piece_id=Z76CLDHUET5KUGMBA2QOF2YYCD5G6KTIY3ZMNU75RTGTACZYATYA, seg_pos={0 0}, upload=14044]: Uploading piece... Aug 7 23:18:39.586534287 (13h39m1.330591157s) [upload=14044]: Next returned nil segment Aug 7 23:18:39.586556352 (13h39m1.330613223s) [upload=14044]: Waiting for error group managing segments... Aug 7 23:18:39.784239406 (13h39m1.528296259s) [noise=true , address=104.36.82.43:4850 , piece_id=NADQYGWW4ZSUM2H7VZOTGBDF7IHDOHN4XWW7PN22LWVN5X36XNNQ, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.789801930 (13h39m1.533858795s) [noise=true , address=104.36.82.21:4487 , piece_id=MMS5K4JLTMWJLIIYSYLW3ALCLESVEYDFL56TLWDXWKWJKWQ2K7HA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.814697149 (13h39m1.55875401s) [noise=true , address=104.36.82.55:4197 , piece_id=QKOQS7SJ2PKH7NX3JHQX27B67GF62HB2GTFSEYHEBUJ3A526SOBQ, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.827998373 (13h39m1.572055234s) [noise=true , address=45.91.116.10:4115 , piece_id=5KYVEX26W4V4YIFPU5Q4PCFNFRROYMCIZ6GX7BGXQ5ZS2VKWPZRQ, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.841687684 (13h39m1.585744538s) [noise=true , address=45.91.116.6:4003 , piece_id=J2MJMYJMZKJFTBDT2DTO43YDBJVUSD2A7HHG3N4GUCRA7BSORCQA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.844299652 (13h39m1.588356512s) [noise=true , address=104.36.82.13:4188 , piece_id=RY6DAZKVGETUITCS2OEQNDKYWHPCLCQ34BADZFGSJLQSE7GJR5SQ, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.880021055 (13h39m1.624077925s) [noise=true , address=104.36.82.5:4863 , piece_id=JAKGK5M3DUOS5QHRDBSU7CKQ5AK3JPJWTMO4FVD77RXS5UXBI3NA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.965854328 (13h39m1.709911181s) [noise=true , address=104.36.82.38:4691 , piece_id=T6ULTGPNZHDN4FIK2QQ4XXPUTXHBYSDG5ZACGYNWGXDSYOJEYXEQ, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:39.968865582 (13h39m1.712922447s) [noise=true , address=156.146.43.225:4449 , piece_id=HUNFO56CIH7SUSQKWDXEKPS5VNOMME7RTFPELRRCJKKCINYMBQYA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:40.001397112 (13h39m1.745453981s) [noise=true , address=45.91.116.9:4269 , piece_id=R5Z23ICCK5IW24TEJFTX4ZV2SFTB2JJOGPWPMJM3KU4TC7F4GO5A, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:40.026713515 (13h39m1.770770363s) [noise=true , address=104.36.82.9:4013 , piece_id=R27TXVFKJWQDJQYGL4GL5LUHO7AIMLICSWY4MUL6XOS7VHLS5KCA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:40.098344358 (13h39m1.842401218s) [noise=true , address=79.127.201.215:4357 , piece_id=Z76CLDHUET5KUGMBA2QOF2YYCD5G6KTIY3ZMNU75RTGTACZYATYA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:40.134297895 (13h39m1.878354754s) [noise=true , address=104.36.82.44:4880 , piece_id=R252YCXF3V3BYKMAMYDQZB62WYCSS2NLI3JTYQPUX5B3NUVMGEMA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:40.277501323 (13h39m2.021558177s) [noise=true , address=79.127.205.246:4265 , piece_id=RYBO73UDU2XDRCKURFIHXCXOTZX7SKCS6Y3TXYNA3HAHBQQXAAQQ, seg_pos={0 0}, upload=14044]: Done uploading piece. err: <nil> Aug 7 23:18:40.277535537 (13h39m2.021592397s) [seg_pos={0 0}, upload=14044]: Segment reached optimal threshold of 14 pieces. Aug 7 23:18:40.277702085 (13h39m2.021758943s) [noise=true , address=104.36.82.47:4552 , piece_id=XMRBJCTRZHKASCQZOG6JJ235AN3LJLQTZ6SRNXGFHJEYRLSRR76A, seg_pos={0 0}, upload=14044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:40.277786501 (13h39m2.021843348s) [noise=true , address=79.127.205.233:4625 , piece_id=2IA76NOFLNV7REJEBJ4Y6OUIS7JZVBCLZAA7HYEG3H3DOUAUTZGA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sq71vD4JQ9eSC57mKXGJDm2rN83CbsByuwi49iESzH4U155kD): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:40.277815673 (13h39m2.021872524s) [noise=true , address=104.36.82.60:4371 , piece_id=I5D4P4SLDOTDK5OP73BOJMOXYZMVDGOSHYOTEXMN2WF6JH6THSVA, seg_pos={0 0}, upload=14044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:40.277834823 (13h39m2.021891669s) [noise=true , address=104.36.82.62:4423 , piece_id=KF2IMNMXFWBVBXEWWXR46CNOHHWUE7IJA2BU7LLWT37GOWOPQNSQ, seg_pos={0 0}, upload=14044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fUb7FPBapbEwdkCE7vsgwcbF5B8KGcDVZAtihDB9SDuvUpzA5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:40.277852230 (13h39m2.021909076s) [seg_pos={0 0}, upload=14044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:18:40.277959476 (13h39m2.022016324s) [upload=14044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:40.277964970 (13h39m2.022021816s) [upload=14044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:48.150909034 (13h39m9.894965884s) [upload=14044]: Done uploading segments... Aug 7 23:18:48.150929677 (13h39m9.894986527s) [upload=14044]: Upload finished. err: <nil> Aug 7 23:19:07.063275110 (13h39m28.80733198s) [upload=14063]: Starting upload Aug 7 23:19:07.063441955 (13h39m28.80749882s) [upload=14063]: Done starting upload Aug 7 23:19:07.063612317 (13h39m28.807669182s) [upload=14063]: Uploading segments... Aug 7 23:19:07.064132300 (13h39m28.808189171s) [upload=14063]: Got next segment. Inline: false Aug 7 23:19:07.064183308 (13h39m28.808240171s) [upload=14063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:07.064189866 (13h39m28.80824673s) [upload=14063]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:20:06.792156395 (13h40m28.536213302s) [upload=14063]: Next segment err: context deadline exceeded Aug 7 23:20:06.792406941 (13h40m28.536463812s) [upload=14063]: Done uploading segments... Aug 7 23:20:06.792438546 (13h40m28.536495408s) [upload=14063]: Upload finished. err: context deadline exceeded Duration: 1m0.000881307s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.348396156s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:20:02 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1.348426363s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................................................XXXXXX.XX.] duration: 2.92520691s FAIL (2025-08-07 23:19:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.925245169s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 830.450566ms FAIL (2025-08-07 23:19:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 830.487209ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [................................................XXXXX.XX.XX.] duration: 924.2937ms FAIL (2025-08-07 23:19:58 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 924.317574ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.................................................XXXXX.XX.XX] duration: 30.000940521s FAIL (2025-08-07 23:19:57 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 30.000971556s core uplink us1

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)