Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.432277381s FAIL (2025-08-07 23:19:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.432306299s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXX...X] duration: 30.000599073s FAIL (2025-08-07 23:19:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000680081s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................X.XXXXX] duration: 1m0.001170504s FAIL (2025-08-07 23:19:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001196657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................................................XXXXXX.XX] duration: 30.291342021s FAIL (2025-08-07 23:19:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.291369714s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....X............X.................] duration: 2.727985784s FAIL (2025-08-07 23:19:35 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 2.728023457s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 17.394118163s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:19:31 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 17.394151691s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [................................................XXX.XXXXXX..] duration: 1.020890846s FAIL (2025-08-07 23:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 48.330046481 (13h32m10.074103328s) [upload=13936]: Uploading segments... Aug 7 23:11:48.330100337 (13h32m10.074157203s) [upload=13936]: Got next segment. Inline: true Aug 7 23:11:48.330105628 (13h32m10.074162488s) [upload=13936]: Waiting for error group managing segments... Aug 7 23:11:48.330144233 (13h32m10.074201079s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.330148601 (13h32m10.074205447s) [upload=13936]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.330151121 (13h32m10.074207967s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:18.060862417 (13h32m39.804919268s) [upload=13936]: Done uploading segments... Aug 7 23:12:18.060910946 (13h32m39.804967806s) [upload=13936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:19.230822944 (13h32m40.974879804s) [upload=13947]: Starting upload Aug 7 23:12:19.230891930 (13h32m40.97494879s) [upload=13947]: Done starting upload Aug 7 23:12:19.230961121 (13h32m40.975017968s) [upload=13947]: Uploading segments... Aug 7 23:12:19.231005268 (13h32m40.975062139s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:19.231009941 (13h32m40.975066801s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:19.231058939 (13h32m40.975115785s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:19.231063293 (13h32m40.975120154s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:19.231065710 (13h32m40.97512257s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:48.962967239 (13h33m10.707024101s) [upload=13947]: Done uploading segments... Aug 7 23:12:48.962986733 (13h33m10.707043592s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:03.240322668 (13h33m24.984379539s) [upload=13961]: Starting upload Aug 7 23:13:03.240467142 (13h33m24.984524008s) [upload=13961]: Done starting upload Aug 7 23:13:03.240603062 (13h33m24.984659911s) [upload=13961]: Uploading segments... Aug 7 23:13:03.240714255 (13h33m24.984771129s) [upload=13961]: Got next segment. Inline: true Aug 7 23:13:03.240723064 (13h33m24.984779933s) [upload=13961]: Waiting for error group managing segments... Aug 7 23:13:03.240804558 (13h33m24.984861428s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:03.240811567 (13h33m24.984868436s) [upload=13961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:03.240816509 (13h33m24.984873376s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:32.970033245 (13h33m54.714090121s) [upload=13961]: Done uploading segments... Aug 7 23:13:32.970064842 (13h33m54.714121693s) [upload=13961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:33.400576508 (13h33m55.144633367s) [upload=13970]: Starting upload Aug 7 23:13:33.400642971 (13h33m55.144699817s) [upload=13970]: Done starting upload Aug 7 23:13:33.400723341 (13h33m55.144780187s) [upload=13970]: Uploading segments... Aug 7 23:13:33.400775896 (13h33m55.144832764s) [upload=13970]: Got next segment. Inline: true Aug 7 23:13:33.400780746 (13h33m55.144837592s) [upload=13970]: Waiting for error group managing segments... Aug 7 23:13:33.400844129 (13h33m55.144900979s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:33.400851223 (13h33m55.144908092s) [upload=13970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:33.400854888 (13h33m55.144911755s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:33.576404182 (13h33m55.320461054s) [upload=13970]: Done uploading segments... Aug 7 23:13:33.576428692 (13h33m55.320485559s) [upload=13970]: Upload finished. err: <nil> Aug 7 23:13:55.968242604 (13h34m17.712299464s) [upload=13977]: Starting upload Aug 7 23:13:55.968304238 (13h34m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.020919967s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X.XX..X..] duration: 13.429031027s FAIL (2025-08-07 23:19:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.429328448s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [..................................................XX..XXXXX.] duration: 845.640066ms FAIL (2025-08-07 23:19:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 845.679126ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 7 out of 60 is failed (threshold=3) [...................................................XX..XXXXX] duration: 1m0.001446799s FAIL (2025-08-07 23:19:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:21.050365938 (13h38m42.79442281s) [upload=14051]: Starting upload Aug 7 23:18:21.050484720 (13h38m42.794541587s) [upload=14051]: Done starting upload Aug 7 23:18:21.050589902 (13h38m42.794646752s) [upload=14051]: Uploading segments... Aug 7 23:18:21.050663797 (13h38m42.794720667s) [upload=14051]: Got next segment. Inline: true Aug 7 23:18:21.050670451 (13h38m42.794727311s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:18:21.050719413 (13h38m42.794776275s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:21.050725163 (13h38m42.794782023s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:21.050728952 (13h38m42.794785811s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:20.780076504 (13h39m42.524133366s) [upload=14051]: Done uploading segments... Aug 7 23:19:20.780099469 (13h39m42.52415633s) [upload=14051]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001467215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..................................................XXXXX.XX.X] duration: 30.002134316s FAIL (2025-08-07 23:19:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.002163708s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.522531576s FAIL (2025-08-07 23:19:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 1.522567351s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.233481462s FAIL (2025-08-07 23:19:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.233521329s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.................................................XXX.XXXXXX.] duration: 863.552598ms FAIL (2025-08-07 23:18:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 48.330046481 (13h32m10.074103328s) [upload=13936]: Uploading segments... Aug 7 23:11:48.330100337 (13h32m10.074157203s) [upload=13936]: Got next segment. Inline: true Aug 7 23:11:48.330105628 (13h32m10.074162488s) [upload=13936]: Waiting for error group managing segments... Aug 7 23:11:48.330144233 (13h32m10.074201079s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.330148601 (13h32m10.074205447s) [upload=13936]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.330151121 (13h32m10.074207967s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:18.060862417 (13h32m39.804919268s) [upload=13936]: Done uploading segments... Aug 7 23:12:18.060910946 (13h32m39.804967806s) [upload=13936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:19.230822944 (13h32m40.974879804s) [upload=13947]: Starting upload Aug 7 23:12:19.230891930 (13h32m40.97494879s) [upload=13947]: Done starting upload Aug 7 23:12:19.230961121 (13h32m40.975017968s) [upload=13947]: Uploading segments... Aug 7 23:12:19.231005268 (13h32m40.975062139s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:19.231009941 (13h32m40.975066801s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:19.231058939 (13h32m40.975115785s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:19.231063293 (13h32m40.975120154s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:19.231065710 (13h32m40.97512257s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:48.962967239 (13h33m10.707024101s) [upload=13947]: Done uploading segments... Aug 7 23:12:48.962986733 (13h33m10.707043592s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:03.240322668 (13h33m24.984379539s) [upload=13961]: Starting upload Aug 7 23:13:03.240467142 (13h33m24.984524008s) [upload=13961]: Done starting upload Aug 7 23:13:03.240603062 (13h33m24.984659911s) [upload=13961]: Uploading segments... Aug 7 23:13:03.240714255 (13h33m24.984771129s) [upload=13961]: Got next segment. Inline: true Aug 7 23:13:03.240723064 (13h33m24.984779933s) [upload=13961]: Waiting for error group managing segments... Aug 7 23:13:03.240804558 (13h33m24.984861428s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:03.240811567 (13h33m24.984868436s) [upload=13961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:03.240816509 (13h33m24.984873376s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:32.970033245 (13h33m54.714090121s) [upload=13961]: Done uploading segments... Aug 7 23:13:32.970064842 (13h33m54.714121693s) [upload=13961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:33.400576508 (13h33m55.144633367s) [upload=13970]: Starting upload Aug 7 23:13:33.400642971 (13h33m55.144699817s) [upload=13970]: Done starting upload Aug 7 23:13:33.400723341 (13h33m55.144780187s) [upload=13970]: Uploading segments... Aug 7 23:13:33.400775896 (13h33m55.144832764s) [upload=13970]: Got next segment. Inline: true Aug 7 23:13:33.400780746 (13h33m55.144837592s) [upload=13970]: Waiting for error group managing segments... Aug 7 23:13:33.400844129 (13h33m55.144900979s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:33.400851223 (13h33m55.144908092s) [upload=13970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:33.400854888 (13h33m55.144911755s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:33.576404182 (13h33m55.320461054s) [upload=13970]: Done uploading segments... Aug 7 23:13:33.576428692 (13h33m55.320485559s) [upload=13970]: Upload finished. err: <nil> Aug 7 23:13:55.968242604 (13h34m17.712299464s) [upload=13977]: Starting upload Aug 7 23:13:55.968304238 (13h34m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Duration: 863.611391ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 50.721545257s FAIL (2025-08-07 23:18:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.537743466 (13h36m37.28180033s) [noise=true , address=104.36.82.58:4295 , piece_id=KR4KLREGCXLEI4IFWJD6YLGLI7Q23LIBTEC7HWVOA7VRFSJXUMTA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.537781852 (13h36m37.281838716s) [noise=true , address=104.36.82.56:4238 , piece_id=4HNVHZOAW7HZ2EOF2P7N2FZSF7ATWKCTRDEES3UFXC64CTDOFFNQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538089856 (13h36m37.282146725s) [noise=true , address=79.127.205.233:4628 , piece_id=FDME537N42NO63U4DYUKZM5LJ4DC3EPFKWNBASKJQPIOWUPLWHSQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538183078 (13h36m37.282239945s) [noise=true , address=104.36.82.18:4384 , piece_id=DYAKHGYSCAYIS74FL5UAQGA7QEAUKQKAUZREL3XPCYMKTGBQJAMQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538558836 (13h36m37.282615698s) [noise=true , address=64.49.13.154:4934 , piece_id=BJFQTBRIUDF5FWEQ6NOWFWRRSPK2XZSO7JFC2OSFJGUYOEU5HESA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538855525 (13h36m37.282912384s) [noise=true , address=104.36.82.15:4276 , piece_id=XYTZKXKAPQAVYMBKBIAEWBWZCPA3N2FGSNYQIB57YW6UFZ5RDFAQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.539228197 (13h36m37.283285059s) [noise=true , address=104.36.82.14:4236 , piece_id=CS3OXUCLQ2DPK3JFITJB6NB5VDG2VVGTYEH727W3LDXJ7LZUTPDQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.539522516 (13h36m37.283579386s) [noise=true , address=104.36.82.37:4654 , piece_id=I4SD4NSZAFBQN2QIKIPODYPAVDU4K5VA33Y5HFMNCR7WBGRSAAFA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.539918855 (13h36m37.283975718s) [noise=true , address=45.91.116.14:4201 , piece_id=F2A32QP2BVAGFDPDEYWS4E3IZVCEJHLQFYG5OWU4NQ66ZW7RTGPA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.540253702 (13h36m37.284310562s) [noise=true , address=104.36.82.45:4915 , piece_id=VLBE5XGBJ5J25WEK3Z5JWT2K4YXTKXM4MKLYFTFADYYVIFT3A5OQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.540567363 (13h36m37.284624223s) [noise=true , address=104.36.82.34:4561 , piece_id=UH2Q3IJ35HC2IZKUAVHLZJA3SP42RTTJZDWHPNXECBSIJSVI5JHA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.540898935 (13h36m37.284955798s) [noise=true , address=104.36.82.44:4901 , piece_id=VUWBGVALZW72CPNZ7ILN53YO32V5L653R5B2K64TZUQLWUMHLMSQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.541250731 (13h36m37.285307609s) [noise=true , address=79.127.220.97:4396 , piece_id=3RJ3I5NCQ2ICIPSLDL5DTQYBMEWLOBRIUWD72FVI23VNRNPA5D3Q, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.541556876 (13h36m37.285613739s) [noise=true , address=104.36.82.16:4316 , piece_id=OZGJRZMEQ5DBSCQZOB7FWQQU2BKGAA6IBGUBW7DMY5C2L3NNNYFA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.541822582 (13h36m37.285879445s) [noise=true , address=79.127.205.234:4594 , piece_id=OFJTZSJ72YSG44L3JW7CL72ZEPB5DIR7YB4WGBBCFFG6BQZW3BDA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.542112356 (13h36m37.286169216s) [noise=true , address=104.36.82.62:4439 , piece_id=EPGZULZ6SSESPBX3W7SA7FBMVVODJYBHFJVVSWW5YWFEX3JPTWUA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:16.116233828 (13h36m37.860290686s) [upload=14015]: Next returned nil segment Aug 7 23:16:16.116258242 (13h36m37.86031511s) [upload=14015]: Waiting for error group managing segments... Aug 7 23:16:16.308361676 (13h36m38.052418631s) [noise=true , address=104.36.82.15:4276 , piece_id=XYTZKXKAPQAVYMBKBIAEWBWZCPA3N2FGSNYQIB57YW6UFZ5RDFAQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.309830241 (13h36m38.053887095s) [noise=true , address=104.36.82.8:4983 , piece_id=WYNILRWHQLFVULBPVMQ57JUYRS4H2FSWHNNSVTTK3SDLUH536UOA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.317473932 (13h36m38.061530786s) [noise=true , address=104.36.82.58:4295 , piece_id=KR4KLREGCXLEI4IFWJD6YLGLI7Q23LIBTEC7HWVOA7VRFSJXUMTA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.351634970 (13h36m38.095691824s) [noise=true , address=45.91.116.14:4201 , piece_id=F2A32QP2BVAGFDPDEYWS4E3IZVCEJHLQFYG5OWU4NQ66ZW7RTGPA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.506107179 (13h36m38.250164029s) [noise=true , address=104.36.82.14:4236 , piece_id=CS3OXUCLQ2DPK3JFITJB6NB5VDG2VVGTYEH727W3LDXJ7LZUTPDQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.520064585 (13h36m38.264121434s) [noise=true , address=104.36.82.18:4384 , piece_id=DYAKHGYSCAYIS74FL5UAQGA7QEAUKQKAUZREL3XPCYMKTGBQJAMQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.546972036 (13h36m38.291028891s) [noise=true , address=104.36.82.45:4915 , piece_id=VLBE5XGBJ5J25WEK3Z5JWT2K4YXTKXM4MKLYFTFADYYVIFT3A5OQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.562591878 (13h36m38.306648732s) [noise=true , address=104.36.82.16:4316 , piece_id=OZGJRZMEQ5DBSCQZOB7FWQQU2BKGAA6IBGUBW7DMY5C2L3NNNYFA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.596229690 (13h36m38.340286536s) [noise=true , address=104.36.82.44:4901 , piece_id=VUWBGVALZW72CPNZ7ILN53YO32V5L653R5B2K64TZUQLWUMHLMSQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.632162612 (13h36m38.376219485s) [noise=true , address=79.127.220.97:4396 , piece_id=3RJ3I5NCQ2ICIPSLDL5DTQYBMEWLOBRIUWD72FVI23VNRNPA5D3Q, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.668777695 (13h36m38.412834553s) [noise=true , address=104.36.82.44:4903 , piece_id=6U4MI3PAL3CJH477KSQL6R72DW4FBH6EDZCZZYCVMJ2ZEES22KCQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.743833653 (13h36m38.487890509s) [noise=true , address=104.36.82.34:4561 , piece_id=UH2Q3IJ35HC2IZKUAVHLZJA3SP42RTTJZDWHPNXECBSIJSVI5JHA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.745433248 (13h36m38.489490098s) [noise=true , address=79.127.205.233:4628 , piece_id=FDME537N42NO63U4DYUKZM5LJ4DC3EPFKWNBASKJQPIOWUPLWHSQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.798587532 (13h36m38.542644378s) [noise=true , address=64.49.13.154:4934 , piece_id=BJFQTBRIUDF5FWEQ6NOWFWRRSPK2XZSO7JFC2OSFJGUYOEU5HESA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.798641902 (13h36m38.542698768s) [seg_pos={0 0}, upload=14015]: Segment reached optimal threshold of 14 pieces. Aug 7 23:16:16.798849571 (13h36m38.542906425s) [noise=true , address=104.36.82.56:4238 , piece_id=4HNVHZOAW7HZ2EOF2P7N2FZSF7ATWKCTRDEES3UFXC64CTDOFFNQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5): protocol: expected piece hash; context canceled; EOF Aug 7 23:16:16.799026289 (13h36m38.543083139s) [noise=true , address=79.127.205.234:4594 , piece_id=OFJTZSJ72YSG44L3JW7CL72ZEPB5DIR7YB4WGBBCFFG6BQZW3BDA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Yerb4RJr5S5Q5G5ZAazicQhTM3MP7JcS1tVjCstAQ18wxP6bp): protocol: expected piece hash; context canceled; EOF Aug 7 23:16:16.799420724 (13h36m38.543477591s) [noise=true , address=104.36.82.62:4439 , piece_id=EPGZULZ6SSESPBX3W7SA7FBMVVODJYBHFJVVSWW5YWFEX3JPTWUA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: write tcp 139.59.144.22:45852->104.36.82.62:4439: use of closed network connection; write tcp 139.59.144.22:45852->104.36.82.62:4439: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45852->104.36.82.62:4439: use of closed network connection Aug 7 23:16:16.799640691 (13h36m38.543697541s) [noise=true , address=104.36.82.37:4654 , piece_id=I4SD4NSZAFBQN2QIKIPODYPAVDU4K5VA33Y5HFMNCR7WBGRSAAFA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 139.59.144.22:44310->104.36.82.37:4654: use of closed network connection; context canceled Aug 7 23:16:16.799682259 (13h36m38.543739129s) [seg_pos={0 0}, upload=14015]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:16:16.799824599 (13h36m38.543881469s) [upload=14015]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:16:16.799831376 (13h36m38.54388825s) [upload=14015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:16.953509647 (13h36m38.69756651s) [upload=14015]: Done uploading segments... Aug 7 23:16:16.953530814 (13h36m38.697587668s) [upload=14015]: Upload finished. err: <nil> Aug 7 23:17:02.195619586 (13h37m23.939676438s) [upload=14027]: Starting upload Aug 7 23:17:02.195703129 (13h37m23.93975998s) [upload=14027]: Done starting upload Aug 7 23:17:02.195969215 (13h37m23.940026065s) [upload=14027]: Uploading segments... Aug 7 23:17:02.196383757 (13h37m23.940440698s) [upload=14027]: Got next segment. Inline: false Aug 7 23:17:02.196513867 (13h37m23.94057073s) [upload=14027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:02.196522322 (13h37m23.940579185s) [upload=14027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:01.919051655 (13h38m23.663108528s) [upload=14027]: Next segment err: context deadline exceeded Aug 7 23:18:01.919296928 (13h38m23.663353793s) [upload=14027]: Done uploading segments... Aug 7 23:18:01.919351614 (13h38m23.663408467s) [upload=14027]: Upload finished. err: context deadline exceeded Duration: 50.721572295s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.890388334s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:18:48 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 3.890408308s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....X............X................] duration: 2.855076288s FAIL (2025-08-07 23:18:45 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 2.855117644s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1m38.227688978s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:18:44 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1m38.227710123s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.180138707s FAIL (2025-08-07 23:18:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.18017497s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.505150002s FAIL (2025-08-07 23:18:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 2.505270742s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXXX.X] duration: 30.000749765s FAIL (2025-08-07 23:18:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000780532s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.....................................................X.XXXX] duration: 1m0.0011869s FAIL (2025-08-07 23:18:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001210922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [..................................................XXX.XXXXXX] duration: 30.001085677s FAIL (2025-08-07 23:18:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 48.330046481 (13h32m10.074103328s) [upload=13936]: Uploading segments... Aug 7 23:11:48.330100337 (13h32m10.074157203s) [upload=13936]: Got next segment. Inline: true Aug 7 23:11:48.330105628 (13h32m10.074162488s) [upload=13936]: Waiting for error group managing segments... Aug 7 23:11:48.330144233 (13h32m10.074201079s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.330148601 (13h32m10.074205447s) [upload=13936]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.330151121 (13h32m10.074207967s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:18.060862417 (13h32m39.804919268s) [upload=13936]: Done uploading segments... Aug 7 23:12:18.060910946 (13h32m39.804967806s) [upload=13936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:19.230822944 (13h32m40.974879804s) [upload=13947]: Starting upload Aug 7 23:12:19.230891930 (13h32m40.97494879s) [upload=13947]: Done starting upload Aug 7 23:12:19.230961121 (13h32m40.975017968s) [upload=13947]: Uploading segments... Aug 7 23:12:19.231005268 (13h32m40.975062139s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:19.231009941 (13h32m40.975066801s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:19.231058939 (13h32m40.975115785s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:19.231063293 (13h32m40.975120154s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:19.231065710 (13h32m40.97512257s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:48.962967239 (13h33m10.707024101s) [upload=13947]: Done uploading segments... Aug 7 23:12:48.962986733 (13h33m10.707043592s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:03.240322668 (13h33m24.984379539s) [upload=13961]: Starting upload Aug 7 23:13:03.240467142 (13h33m24.984524008s) [upload=13961]: Done starting upload Aug 7 23:13:03.240603062 (13h33m24.984659911s) [upload=13961]: Uploading segments... Aug 7 23:13:03.240714255 (13h33m24.984771129s) [upload=13961]: Got next segment. Inline: true Aug 7 23:13:03.240723064 (13h33m24.984779933s) [upload=13961]: Waiting for error group managing segments... Aug 7 23:13:03.240804558 (13h33m24.984861428s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:03.240811567 (13h33m24.984868436s) [upload=13961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:03.240816509 (13h33m24.984873376s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:32.970033245 (13h33m54.714090121s) [upload=13961]: Done uploading segments... Aug 7 23:13:32.970064842 (13h33m54.714121693s) [upload=13961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:33.400576508 (13h33m55.144633367s) [upload=13970]: Starting upload Aug 7 23:13:33.400642971 (13h33m55.144699817s) [upload=13970]: Done starting upload Aug 7 23:13:33.400723341 (13h33m55.144780187s) [upload=13970]: Uploading segments... Aug 7 23:13:33.400775896 (13h33m55.144832764s) [upload=13970]: Got next segment. Inline: true Aug 7 23:13:33.400780746 (13h33m55.144837592s) [upload=13970]: Waiting for error group managing segments... Aug 7 23:13:33.400844129 (13h33m55.144900979s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:33.400851223 (13h33m55.144908092s) [upload=13970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:33.400854888 (13h33m55.144911755s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:33.576404182 (13h33m55.320461054s) [upload=13970]: Done uploading segments... Aug 7 23:13:33.576428692 (13h33m55.320485559s) [upload=13970]: Upload finished. err: <nil> Aug 7 23:13:55.968242604 (13h34m17.712299464s) [upload=13977]: Starting upload Aug 7 23:13:55.968304238 (13h34m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:18:01.235662578 (13h38m22.979719439s) [upload=14043]: Starting upload Aug 7 23:18:01.235769067 (13h38m22.97982594s) [upload=14043]: Done starting upload Aug 7 23:18:01.235873103 (13h38m22.979929971s) [upload=14043]: Uploading segments... Aug 7 23:18:01.235955349 (13h38m22.98001222s) [upload=14043]: Got next segment. Inline: true Aug 7 23:18:01.235976122 (13h38m22.980032984s) [upload=14043]: Waiting for error group managing segments... Aug 7 23:18:01.236032940 (13h38m22.980089801s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:01.236038848 (13h38m22.980095712s) [upload=14043]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:01.236056240 (13h38m22.980113107s) [upload=14043]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:30.962606072 (13h38m52.706662932s) [upload=14043]: Done uploading segments... Aug 7 23:18:30.962671417 (13h38m52.706728279s) [upload=14043]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00120571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................................................XX..XXXX] duration: 1m0.001516613s FAIL (2025-08-07 23:18:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 885]: Starting upload Aug 7 23:08:53.467436784 (13h29m15.211493636s) [upload=13885]: Done starting upload Aug 7 23:08:53.467521314 (13h29m15.211578165s) [upload=13885]: Uploading segments... Aug 7 23:08:53.467570153 (13h29m15.211627016s) [upload=13885]: Got next segment. Inline: true Aug 7 23:08:53.467576092 (13h29m15.211632953s) [upload=13885]: Waiting for error group managing segments... Aug 7 23:08:53.467621884 (13h29m15.211678733s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:53.467627561 (13h29m15.211684422s) [upload=13885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:53.467631218 (13h29m15.211688072s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:53.630342089 (13h29m15.374398952s) [upload=13885]: Done uploading segments... Aug 7 23:08:53.630366467 (13h29m15.37442333s) [upload=13885]: Upload finished. err: <nil> Aug 7 23:10:03.085243036 (13h30m24.829299916s) [upload=13904]: Starting upload Aug 7 23:10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:20.916777280 (13h37m42.660834146s) [upload=14032]: Starting upload Aug 7 23:17:20.916877286 (13h37m42.660934149s) [upload=14032]: Done starting upload Aug 7 23:17:20.917002557 (13h37m42.661059408s) [upload=14032]: Uploading segments... Aug 7 23:17:20.917058206 (13h37m42.661115077s) [upload=14032]: Got next segment. Inline: true Aug 7 23:17:20.917064317 (13h37m42.661121178s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:17:20.917116347 (13h37m42.661173214s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:20.917123150 (13h37m42.661180014s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:20.917127779 (13h37m42.661184648s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:20.640884349 (13h38m42.384941214s) [upload=14032]: Done uploading segments... Aug 7 23:18:20.640914749 (13h38m42.384971617s) [upload=14032]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001534701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...................................................XXXXX.XX.] duration: 968.9036ms FAIL (2025-08-07 23:18:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 968.950613ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.000736164s FAIL (2025-08-07 23:18:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== HWELIBFLJC6K2IPVV2J5U7O55IKOBUOLLHLMVBFG6AVIEXMP24LA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.788084295 (13h32m29.532141146s) [noise=true , address=77.90.31.11:4554 , piece_id=PMGZKAJ2VRGR5H3KQBKL5SHCGSU5DRDGEZBNPKI7KFNO5AHHNKCA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.790692188 (13h32m29.534749035s) [noise=true , address=77.90.31.19:4841 , piece_id=BCXQUB2X5VVIFCHUAHQGD43UAMNTNBPO2ZMZ6LF4TPZBMBER2NJA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.827285350 (13h32m29.571342201s) [noise=true , address=77.90.31.17:4772 , piece_id=JHDYI7MUFLJRWVH3IUOIKHAIAUPNIQNDTRTWOIBH7QU4T5NH6RZQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.827336330 (13h32m29.57139319s) [seg_pos={0 0}, upload=13939]: Segment reached optimal threshold of 13 pieces. Aug 7 23:12:07.827615018 (13h32m29.571671881s) [noise=true , address=45.91.116.13:4154 , piece_id=OYQRDW6EPXAONOYTIGW4HQTXBUH53VSM4IVIYMKQOPNMGPOHOR2Q, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 23:12:07.827796613 (13h32m29.571853464s) [noise=true , address=104.36.82.8:4969 , piece_id=7MQR2SVUM76G22K6RUTCJKMERMX4HLX4EAMZJMZS6D3UVYYHYCVQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection Aug 7 23:12:07.827901392 (13h32m29.571958243s) [noise=true , address=104.36.82.2:4730 , piece_id=GZPRAOONQUIG43X5ODD2F22NNMQWCOYUIU66LL6SWJWVWTZ6R36Q, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 139.59.144.22:60032->104.36.82.2:4730: use of closed network connection; context canceled Aug 7 23:12:07.827937880 (13h32m29.571994732s) [seg_pos={0 0}, upload=13939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:12:07.828022548 (13h32m29.572079408s) [upload=13939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:12:07.828027828 (13h32m29.572084689s) [upload=13939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:07.965146491 (13h32m29.709203382s) [upload=13939]: Done uploading segments... Aug 7 23:12:07.965172031 (13h32m29.709228879s) [upload=13939]: Upload finished. err: <nil> Aug 7 23:12:24.531927541 (13h32m46.275984394s) [upload=13949]: Starting upload Aug 7 23:12:24.532011658 (13h32m46.276068518s) [upload=13949]: Done starting upload Aug 7 23:12:24.532148550 (13h32m46.276205412s) [upload=13949]: Uploading segments... Aug 7 23:12:24.532297280 (13h32m46.27635414s) [upload=13949]: Got next segment. Inline: false Aug 7 23:12:24.532373131 (13h32m46.27643s) [upload=13949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:24.532381517 (13h32m46.276438387s) [upload=13949]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:12:54.256553034 (13h33m16.000609899s) [upload=13949]: Next segment err: context deadline exceeded Aug 7 23:12:54.256729208 (13h33m16.000786059s) [upload=13949]: Done uploading segments... Aug 7 23:12:54.256746630 (13h33m16.000803484s) [upload=13949]: Upload finished. err: context deadline exceeded Aug 7 23:13:11.095725057 (13h33m32.839781903s) [upload=13963]: Starting upload Aug 7 23:13:11.095819942 (13h33m32.839876788s) [upload=13963]: Done starting upload Aug 7 23:13:11.096132701 (13h33m32.84018955s) [upload=13963]: Uploading segments... Aug 7 23:13:11.096178064 (13h33m32.840234933s) [upload=13963]: Got next segment. Inline: false Aug 7 23:13:11.096234910 (13h33m32.840291778s) [upload=13963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:11.096241955 (13h33m32.840298916s) [upload=13963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:40.822883262 (13h34m2.566940132s) [upload=13963]: Next segment err: context deadline exceeded Aug 7 23:13:40.822962100 (13h34m2.567018951s) [upload=13963]: Done uploading segments... Aug 7 23:13:40.823009194 (13h34m2.567066056s) [upload=13963]: Upload finished. err: context deadline exceeded Aug 7 23:13:54.465645519 (13h34m16.209702379s) [upload=13976]: Starting upload Aug 7 23:13:54.465726456 (13h34m16.209783322s) [upload=13976]: Done starting upload Aug 7 23:13:54.465831098 (13h34m16.20988796s) [upload=13976]: Uploading segments... Aug 7 23:13:54.466125339 (13h34m16.210182189s) [upload=13976]: Got next segment. Inline: false Aug 7 23:13:54.466207955 (13h34m16.210264835s) [upload=13976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:54.466231453 (13h34m16.21028832s) [upload=13976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:24.194723155 (13h34m45.938780024s) [upload=13976]: Next segment err: context deadline exceeded Aug 7 23:14:24.194812649 (13h34m45.938869515s) [upload=13976]: Done uploading segments... Aug 7 23:14:24.194846807 (13h34m45.938903668s) [upload=13976]: Upload finished. err: context deadline exceeded Aug 7 23:14:24.595327088 (13h34m46.339383935s) [upload=13984]: Starting upload Aug 7 23:14:24.595396123 (13h34m46.339452983s) [upload=13984]: Done starting upload Aug 7 23:14:24.595522981 (13h34m46.339579831s) [upload=13984]: Uploading segments... Aug 7 23:14:24.595654427 (13h34m46.339711274s) [upload=13984]: Got next segment. Inline: false Aug 7 23:14:24.595690189 (13h34m46.339747048s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:24.595695677 (13h34m46.339752537s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:54.326037198 (13h35m16.070094059s) [upload=13984]: Next segment err: context deadline exceeded Aug 7 23:14:54.326107986 (13h35m16.070164835s) [upload=13984]: Done uploading segments... Aug 7 23:14:54.326129117 (13h35m16.07018597s) [upload=13984]: Upload finished. err: context deadline exceeded Aug 7 23:15:12.257450784 (13h35m34.001507635s) [upload=13996]: Starting upload Aug 7 23:15:12.257569128 (13h35m34.00162599s) [upload=13996]: Done starting upload Aug 7 23:15:12.257701999 (13h35m34.001758859s) [upload=13996]: Uploading segments... Aug 7 23:15:12.258022354 (13h35m34.002079211s) [upload=13996]: Got next segment. Inline: false Aug 7 23:15:12.258091150 (13h35m34.002148s) [upload=13996]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:12.258097670 (13h35m34.002154531s) [upload=13996]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:41.986853387 (13h36m3.73091025s) [upload=13996]: Next segment err: context deadline exceeded Aug 7 23:15:41.986874552 (13h36m3.730931402s) [upload=13996]: Done uploading segments... Aug 7 23:15:41.986894621 (13h36m3.730951471s) [upload=13996]: Upload finished. err: context deadline exceeded Aug 7 23:16:09.685754413 (13h36m31.429811283s) [upload=14012]: Starting upload Aug 7 23:16:09.685841115 (13h36m31.429897983s) [upload=14012]: Done starting upload Aug 7 23:16:09.686066017 (13h36m31.430122878s) [upload=14012]: Uploading segments... Aug 7 23:16:09.686183581 (13h36m31.430240431s) [upload=14012]: Got next segment. Inline: false Aug 7 23:16:09.686224472 (13h36m31.43028132s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.686231334 (13h36m31.430288193s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:39.416007056 (13h37m1.160063926s) [upload=14012]: Next segment err: context deadline exceeded Aug 7 23:16:39.416091190 (13h37m1.160148037s) [upload=14012]: Done uploading segments... Aug 7 23:16:39.416112103 (13h37m1.160168951s) [upload=14012]: Upload finished. err: context deadline exceeded Aug 7 23:17:19.134115876 (13h37m40.87817276s) [upload=14031]: Starting upload Aug 7 23:17:19.134217017 (13h37m40.87827389s) [upload=14031]: Done starting upload Aug 7 23:17:19.134369000 (13h37m40.878425864s) [upload=14031]: Uploading segments... Aug 7 23:17:19.134768595 (13h37m40.878825466s) [upload=14031]: Got next segment. Inline: false Aug 7 23:17:19.134850804 (13h37m40.878907676s) [upload=14031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:19.134856837 (13h37m40.878913708s) [upload=14031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:17:48.861585973 (13h38m10.605642848s) [upload=14031]: Next segment err: context deadline exceeded Aug 7 23:17:48.861771277 (13h38m10.605828148s) [upload=14031]: Done uploading segments... Aug 7 23:17:48.861795148 (13h38m10.605852011s) [upload=14031]: Upload finished. err: context deadline exceeded Aug 7 23:17:49.307656085 (13h38m11.051712931s) [upload=14039]: Starting upload Aug 7 23:17:49.307705424 (13h38m11.051762272s) [upload=14039]: Done starting upload Aug 7 23:17:49.308016567 (13h38m11.052073415s) [upload=14039]: Uploading segments... Aug 7 23:17:49.308064138 (13h38m11.052120985s) [upload=14039]: Got next segment. Inline: false Aug 7 23:17:49.308123883 (13h38m11.052180743s) [upload=14039]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:49.308129375 (13h38m11.052186235s) [upload=14039]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:19.032758502 (13h38m40.776815367s) [upload=14039]: Next segment err: context deadline exceeded Aug 7 23:18:19.032971931 (13h38m40.777028782s) [upload=14039]: Done uploading segments... Aug 7 23:18:19.033014318 (13h38m40.777071179s) [upload=14039]: Upload finished. err: context deadline exceeded Duration: 30.000755237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 1.415173252s FAIL (2025-08-07 23:18:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.415275442s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X.XX..X.] duration: 2.974506886s FAIL (2025-08-07 23:18:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.974543294s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.141633115s FAIL (2025-08-07 23:18:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.141748403s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 1m0.001415737s FAIL (2025-08-07 23:18:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.537743466 (13h36m37.28180033s) [noise=true , address=104.36.82.58:4295 , piece_id=KR4KLREGCXLEI4IFWJD6YLGLI7Q23LIBTEC7HWVOA7VRFSJXUMTA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.537781852 (13h36m37.281838716s) [noise=true , address=104.36.82.56:4238 , piece_id=4HNVHZOAW7HZ2EOF2P7N2FZSF7ATWKCTRDEES3UFXC64CTDOFFNQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538089856 (13h36m37.282146725s) [noise=true , address=79.127.205.233:4628 , piece_id=FDME537N42NO63U4DYUKZM5LJ4DC3EPFKWNBASKJQPIOWUPLWHSQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538183078 (13h36m37.282239945s) [noise=true , address=104.36.82.18:4384 , piece_id=DYAKHGYSCAYIS74FL5UAQGA7QEAUKQKAUZREL3XPCYMKTGBQJAMQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538558836 (13h36m37.282615698s) [noise=true , address=64.49.13.154:4934 , piece_id=BJFQTBRIUDF5FWEQ6NOWFWRRSPK2XZSO7JFC2OSFJGUYOEU5HESA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.538855525 (13h36m37.282912384s) [noise=true , address=104.36.82.15:4276 , piece_id=XYTZKXKAPQAVYMBKBIAEWBWZCPA3N2FGSNYQIB57YW6UFZ5RDFAQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.539228197 (13h36m37.283285059s) [noise=true , address=104.36.82.14:4236 , piece_id=CS3OXUCLQ2DPK3JFITJB6NB5VDG2VVGTYEH727W3LDXJ7LZUTPDQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.539522516 (13h36m37.283579386s) [noise=true , address=104.36.82.37:4654 , piece_id=I4SD4NSZAFBQN2QIKIPODYPAVDU4K5VA33Y5HFMNCR7WBGRSAAFA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.539918855 (13h36m37.283975718s) [noise=true , address=45.91.116.14:4201 , piece_id=F2A32QP2BVAGFDPDEYWS4E3IZVCEJHLQFYG5OWU4NQ66ZW7RTGPA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.540253702 (13h36m37.284310562s) [noise=true , address=104.36.82.45:4915 , piece_id=VLBE5XGBJ5J25WEK3Z5JWT2K4YXTKXM4MKLYFTFADYYVIFT3A5OQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.540567363 (13h36m37.284624223s) [noise=true , address=104.36.82.34:4561 , piece_id=UH2Q3IJ35HC2IZKUAVHLZJA3SP42RTTJZDWHPNXECBSIJSVI5JHA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.540898935 (13h36m37.284955798s) [noise=true , address=104.36.82.44:4901 , piece_id=VUWBGVALZW72CPNZ7ILN53YO32V5L653R5B2K64TZUQLWUMHLMSQ, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.541250731 (13h36m37.285307609s) [noise=true , address=79.127.220.97:4396 , piece_id=3RJ3I5NCQ2ICIPSLDL5DTQYBMEWLOBRIUWD72FVI23VNRNPA5D3Q, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.541556876 (13h36m37.285613739s) [noise=true , address=104.36.82.16:4316 , piece_id=OZGJRZMEQ5DBSCQZOB7FWQQU2BKGAA6IBGUBW7DMY5C2L3NNNYFA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.541822582 (13h36m37.285879445s) [noise=true , address=79.127.205.234:4594 , piece_id=OFJTZSJ72YSG44L3JW7CL72ZEPB5DIR7YB4WGBBCFFG6BQZW3BDA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:15.542112356 (13h36m37.286169216s) [noise=true , address=104.36.82.62:4439 , piece_id=EPGZULZ6SSESPBX3W7SA7FBMVVODJYBHFJVVSWW5YWFEX3JPTWUA, seg_pos={0 0}, upload=14015]: Uploading piece... Aug 7 23:16:16.116233828 (13h36m37.860290686s) [upload=14015]: Next returned nil segment Aug 7 23:16:16.116258242 (13h36m37.86031511s) [upload=14015]: Waiting for error group managing segments... Aug 7 23:16:16.308361676 (13h36m38.052418631s) [noise=true , address=104.36.82.15:4276 , piece_id=XYTZKXKAPQAVYMBKBIAEWBWZCPA3N2FGSNYQIB57YW6UFZ5RDFAQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.309830241 (13h36m38.053887095s) [noise=true , address=104.36.82.8:4983 , piece_id=WYNILRWHQLFVULBPVMQ57JUYRS4H2FSWHNNSVTTK3SDLUH536UOA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.317473932 (13h36m38.061530786s) [noise=true , address=104.36.82.58:4295 , piece_id=KR4KLREGCXLEI4IFWJD6YLGLI7Q23LIBTEC7HWVOA7VRFSJXUMTA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.351634970 (13h36m38.095691824s) [noise=true , address=45.91.116.14:4201 , piece_id=F2A32QP2BVAGFDPDEYWS4E3IZVCEJHLQFYG5OWU4NQ66ZW7RTGPA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.506107179 (13h36m38.250164029s) [noise=true , address=104.36.82.14:4236 , piece_id=CS3OXUCLQ2DPK3JFITJB6NB5VDG2VVGTYEH727W3LDXJ7LZUTPDQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.520064585 (13h36m38.264121434s) [noise=true , address=104.36.82.18:4384 , piece_id=DYAKHGYSCAYIS74FL5UAQGA7QEAUKQKAUZREL3XPCYMKTGBQJAMQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.546972036 (13h36m38.291028891s) [noise=true , address=104.36.82.45:4915 , piece_id=VLBE5XGBJ5J25WEK3Z5JWT2K4YXTKXM4MKLYFTFADYYVIFT3A5OQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.562591878 (13h36m38.306648732s) [noise=true , address=104.36.82.16:4316 , piece_id=OZGJRZMEQ5DBSCQZOB7FWQQU2BKGAA6IBGUBW7DMY5C2L3NNNYFA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.596229690 (13h36m38.340286536s) [noise=true , address=104.36.82.44:4901 , piece_id=VUWBGVALZW72CPNZ7ILN53YO32V5L653R5B2K64TZUQLWUMHLMSQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.632162612 (13h36m38.376219485s) [noise=true , address=79.127.220.97:4396 , piece_id=3RJ3I5NCQ2ICIPSLDL5DTQYBMEWLOBRIUWD72FVI23VNRNPA5D3Q, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.668777695 (13h36m38.412834553s) [noise=true , address=104.36.82.44:4903 , piece_id=6U4MI3PAL3CJH477KSQL6R72DW4FBH6EDZCZZYCVMJ2ZEES22KCQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.743833653 (13h36m38.487890509s) [noise=true , address=104.36.82.34:4561 , piece_id=UH2Q3IJ35HC2IZKUAVHLZJA3SP42RTTJZDWHPNXECBSIJSVI5JHA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.745433248 (13h36m38.489490098s) [noise=true , address=79.127.205.233:4628 , piece_id=FDME537N42NO63U4DYUKZM5LJ4DC3EPFKWNBASKJQPIOWUPLWHSQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.798587532 (13h36m38.542644378s) [noise=true , address=64.49.13.154:4934 , piece_id=BJFQTBRIUDF5FWEQ6NOWFWRRSPK2XZSO7JFC2OSFJGUYOEU5HESA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: <nil> Aug 7 23:16:16.798641902 (13h36m38.542698768s) [seg_pos={0 0}, upload=14015]: Segment reached optimal threshold of 14 pieces. Aug 7 23:16:16.798849571 (13h36m38.542906425s) [noise=true , address=104.36.82.56:4238 , piece_id=4HNVHZOAW7HZ2EOF2P7N2FZSF7ATWKCTRDEES3UFXC64CTDOFFNQ, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5): protocol: expected piece hash; context canceled; EOF Aug 7 23:16:16.799026289 (13h36m38.543083139s) [noise=true , address=79.127.205.234:4594 , piece_id=OFJTZSJ72YSG44L3JW7CL72ZEPB5DIR7YB4WGBBCFFG6BQZW3BDA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Yerb4RJr5S5Q5G5ZAazicQhTM3MP7JcS1tVjCstAQ18wxP6bp): protocol: expected piece hash; context canceled; EOF Aug 7 23:16:16.799420724 (13h36m38.543477591s) [noise=true , address=104.36.82.62:4439 , piece_id=EPGZULZ6SSESPBX3W7SA7FBMVVODJYBHFJVVSWW5YWFEX3JPTWUA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: write tcp 139.59.144.22:45852->104.36.82.62:4439: use of closed network connection; write tcp 139.59.144.22:45852->104.36.82.62:4439: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45852->104.36.82.62:4439: use of closed network connection Aug 7 23:16:16.799640691 (13h36m38.543697541s) [noise=true , address=104.36.82.37:4654 , piece_id=I4SD4NSZAFBQN2QIKIPODYPAVDU4K5VA33Y5HFMNCR7WBGRSAAFA, seg_pos={0 0}, upload=14015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 139.59.144.22:44310->104.36.82.37:4654: use of closed network connection; context canceled Aug 7 23:16:16.799682259 (13h36m38.543739129s) [seg_pos={0 0}, upload=14015]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:16:16.799824599 (13h36m38.543881469s) [upload=14015]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:16:16.799831376 (13h36m38.54388825s) [upload=14015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:16.953509647 (13h36m38.69756651s) [upload=14015]: Done uploading segments... Aug 7 23:16:16.953530814 (13h36m38.697587668s) [upload=14015]: Upload finished. err: <nil> Aug 7 23:17:02.195619586 (13h37m23.939676438s) [upload=14027]: Starting upload Aug 7 23:17:02.195703129 (13h37m23.93975998s) [upload=14027]: Done starting upload Aug 7 23:17:02.195969215 (13h37m23.940026065s) [upload=14027]: Uploading segments... Aug 7 23:17:02.196383757 (13h37m23.940440698s) [upload=14027]: Got next segment. Inline: false Aug 7 23:17:02.196513867 (13h37m23.94057073s) [upload=14027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:02.196522322 (13h37m23.940579185s) [upload=14027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:01.919051655 (13h38m23.663108528s) [upload=14027]: Next segment err: context deadline exceeded Aug 7 23:18:01.919296928 (13h38m23.663353793s) [upload=14027]: Done uploading segments... Aug 7 23:18:01.919351614 (13h38m23.663408467s) [upload=14027]: Upload finished. err: context deadline exceeded Duration: 1m0.001446442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXX.XX] duration: 30.00146214s FAIL (2025-08-07 23:17:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001505498s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [...................................................XXX.XXXXX] duration: 30.001376066s FAIL (2025-08-07 23:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 23:11:16.095421525 (13h31m37.839478385s) [upload=13924]: Uploading segments... Aug 7 23:11:16.095460332 (13h31m37.839517201s) [upload=13924]: Got next segment. Inline: true Aug 7 23:11:16.095465792 (13h31m37.839522653s) [upload=13924]: Waiting for error group managing segments... Aug 7 23:11:16.095512445 (13h31m37.839569308s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:16.095517610 (13h31m37.839574471s) [upload=13924]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:16.095521136 (13h31m37.839577996s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:16.231539768 (13h31m37.975596629s) [upload=13924]: Done uploading segments... Aug 7 23:11:16.231558376 (13h31m37.975615223s) [upload=13924]: Upload finished. err: <nil> Aug 7 23:11:48.329873818 (13h32m10.073930665s) [upload=13936]: Starting upload Aug 7 23:11:48.329969166 (13h32m10.074026026s) [upload=13936]: Done starting upload Aug 7 23:11:48.330046481 (13h32m10.074103328s) [upload=13936]: Uploading segments... Aug 7 23:11:48.330100337 (13h32m10.074157203s) [upload=13936]: Got next segment. Inline: true Aug 7 23:11:48.330105628 (13h32m10.074162488s) [upload=13936]: Waiting for error group managing segments... Aug 7 23:11:48.330144233 (13h32m10.074201079s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.330148601 (13h32m10.074205447s) [upload=13936]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.330151121 (13h32m10.074207967s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:18.060862417 (13h32m39.804919268s) [upload=13936]: Done uploading segments... Aug 7 23:12:18.060910946 (13h32m39.804967806s) [upload=13936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:19.230822944 (13h32m40.974879804s) [upload=13947]: Starting upload Aug 7 23:12:19.230891930 (13h32m40.97494879s) [upload=13947]: Done starting upload Aug 7 23:12:19.230961121 (13h32m40.975017968s) [upload=13947]: Uploading segments... Aug 7 23:12:19.231005268 (13h32m40.975062139s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:19.231009941 (13h32m40.975066801s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:19.231058939 (13h32m40.975115785s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:19.231063293 (13h32m40.975120154s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:19.231065710 (13h32m40.97512257s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:48.962967239 (13h33m10.707024101s) [upload=13947]: Done uploading segments... Aug 7 23:12:48.962986733 (13h33m10.707043592s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:03.240322668 (13h33m24.984379539s) [upload=13961]: Starting upload Aug 7 23:13:03.240467142 (13h33m24.984524008s) [upload=13961]: Done starting upload Aug 7 23:13:03.240603062 (13h33m24.984659911s) [upload=13961]: Uploading segments... Aug 7 23:13:03.240714255 (13h33m24.984771129s) [upload=13961]: Got next segment. Inline: true Aug 7 23:13:03.240723064 (13h33m24.984779933s) [upload=13961]: Waiting for error group managing segments... Aug 7 23:13:03.240804558 (13h33m24.984861428s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:03.240811567 (13h33m24.984868436s) [upload=13961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:03.240816509 (13h33m24.984873376s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:32.970033245 (13h33m54.714090121s) [upload=13961]: Done uploading segments... Aug 7 23:13:32.970064842 (13h33m54.714121693s) [upload=13961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:33.400576508 (13h33m55.144633367s) [upload=13970]: Starting upload Aug 7 23:13:33.400642971 (13h33m55.144699817s) [upload=13970]: Done starting upload Aug 7 23:13:33.400723341 (13h33m55.144780187s) [upload=13970]: Uploading segments... Aug 7 23:13:33.400775896 (13h33m55.144832764s) [upload=13970]: Got next segment. Inline: true Aug 7 23:13:33.400780746 (13h33m55.144837592s) [upload=13970]: Waiting for error group managing segments... Aug 7 23:13:33.400844129 (13h33m55.144900979s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:33.400851223 (13h33m55.144908092s) [upload=13970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:33.400854888 (13h33m55.144911755s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:33.576404182 (13h33m55.320461054s) [upload=13970]: Done uploading segments... Aug 7 23:13:33.576428692 (13h33m55.320485559s) [upload=13970]: Upload finished. err: <nil> Aug 7 23:13:55.968242604 (13h34m17.712299464s) [upload=13977]: Starting upload Aug 7 23:13:55.968304238 (13h34m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:23.340430421 (13h37m45.084487291s) [upload=14034]: Starting upload Aug 7 23:17:23.340585398 (13h37m45.084642265s) [upload=14034]: Done starting upload Aug 7 23:17:23.340743782 (13h37m45.084800644s) [upload=14034]: Uploading segments... Aug 7 23:17:23.340847668 (13h37m45.084904539s) [upload=14034]: Got next segment. Inline: true Aug 7 23:17:23.340857086 (13h37m45.084913958s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:17:23.340963706 (13h37m45.085020572s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:23.340973939 (13h37m45.085031006s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:23.340979371 (13h37m45.08503624s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.063547301 (13h38m14.807604164s) [upload=14034]: Done uploading segments... Aug 7 23:17:53.063575766 (13h38m14.807632627s) [upload=14034]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001404437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.310666537s FAIL (2025-08-07 23:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 1.310699817s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration: 2.944529242s FAIL (2025-08-07 23:17:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 2.944549201s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.000900551s FAIL (2025-08-07 23:17:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== , piece_id=IO7M6EL3TAE6WPD6WNKQMU7JYHY4OL3E4E46JNEXVKQQGSWDMNSQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.691257838 (13h32m29.435314704s) [noise=true , address=45.91.116.7:4010 , piece_id=KGKZWJIJRCKP2KAM2BXH5EQXZU6AY4SKU3YIJTUIMPEJMHBSBTXA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.698168459 (13h32m29.442225309s) [noise=true , address=45.91.116.9:4261 , piece_id=YQSAMMSUMO6D6FISYIXASKMTXBUNN3HTICAJSI5MIQCAXC72TY7A, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.708048205 (13h32m29.452105056s) [noise=true , address=104.36.82.6:4876 , piece_id=Y2HFXOS6O7DOGA57LYIQCTHG46PF5UILJGF2VJRJ2XXYUZBGLSCA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.724323502 (13h32m29.468380349s) [noise=true , address=77.90.31.14:4661 , piece_id=HWELIBFLJC6K2IPVV2J5U7O55IKOBUOLLHLMVBFG6AVIEXMP24LA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.788084295 (13h32m29.532141146s) [noise=true , address=77.90.31.11:4554 , piece_id=PMGZKAJ2VRGR5H3KQBKL5SHCGSU5DRDGEZBNPKI7KFNO5AHHNKCA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.790692188 (13h32m29.534749035s) [noise=true , address=77.90.31.19:4841 , piece_id=BCXQUB2X5VVIFCHUAHQGD43UAMNTNBPO2ZMZ6LF4TPZBMBER2NJA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.827285350 (13h32m29.571342201s) [noise=true , address=77.90.31.17:4772 , piece_id=JHDYI7MUFLJRWVH3IUOIKHAIAUPNIQNDTRTWOIBH7QU4T5NH6RZQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.827336330 (13h32m29.57139319s) [seg_pos={0 0}, upload=13939]: Segment reached optimal threshold of 13 pieces. Aug 7 23:12:07.827615018 (13h32m29.571671881s) [noise=true , address=45.91.116.13:4154 , piece_id=OYQRDW6EPXAONOYTIGW4HQTXBUH53VSM4IVIYMKQOPNMGPOHOR2Q, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 23:12:07.827796613 (13h32m29.571853464s) [noise=true , address=104.36.82.8:4969 , piece_id=7MQR2SVUM76G22K6RUTCJKMERMX4HLX4EAMZJMZS6D3UVYYHYCVQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection Aug 7 23:12:07.827901392 (13h32m29.571958243s) [noise=true , address=104.36.82.2:4730 , piece_id=GZPRAOONQUIG43X5ODD2F22NNMQWCOYUIU66LL6SWJWVWTZ6R36Q, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 139.59.144.22:60032->104.36.82.2:4730: use of closed network connection; context canceled Aug 7 23:12:07.827937880 (13h32m29.571994732s) [seg_pos={0 0}, upload=13939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:12:07.828022548 (13h32m29.572079408s) [upload=13939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:12:07.828027828 (13h32m29.572084689s) [upload=13939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:07.965146491 (13h32m29.709203382s) [upload=13939]: Done uploading segments... Aug 7 23:12:07.965172031 (13h32m29.709228879s) [upload=13939]: Upload finished. err: <nil> Aug 7 23:12:24.531927541 (13h32m46.275984394s) [upload=13949]: Starting upload Aug 7 23:12:24.532011658 (13h32m46.276068518s) [upload=13949]: Done starting upload Aug 7 23:12:24.532148550 (13h32m46.276205412s) [upload=13949]: Uploading segments... Aug 7 23:12:24.532297280 (13h32m46.27635414s) [upload=13949]: Got next segment. Inline: false Aug 7 23:12:24.532373131 (13h32m46.27643s) [upload=13949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:24.532381517 (13h32m46.276438387s) [upload=13949]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:12:54.256553034 (13h33m16.000609899s) [upload=13949]: Next segment err: context deadline exceeded Aug 7 23:12:54.256729208 (13h33m16.000786059s) [upload=13949]: Done uploading segments... Aug 7 23:12:54.256746630 (13h33m16.000803484s) [upload=13949]: Upload finished. err: context deadline exceeded Aug 7 23:13:11.095725057 (13h33m32.839781903s) [upload=13963]: Starting upload Aug 7 23:13:11.095819942 (13h33m32.839876788s) [upload=13963]: Done starting upload Aug 7 23:13:11.096132701 (13h33m32.84018955s) [upload=13963]: Uploading segments... Aug 7 23:13:11.096178064 (13h33m32.840234933s) [upload=13963]: Got next segment. Inline: false Aug 7 23:13:11.096234910 (13h33m32.840291778s) [upload=13963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:11.096241955 (13h33m32.840298916s) [upload=13963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:40.822883262 (13h34m2.566940132s) [upload=13963]: Next segment err: context deadline exceeded Aug 7 23:13:40.822962100 (13h34m2.567018951s) [upload=13963]: Done uploading segments... Aug 7 23:13:40.823009194 (13h34m2.567066056s) [upload=13963]: Upload finished. err: context deadline exceeded Aug 7 23:13:54.465645519 (13h34m16.209702379s) [upload=13976]: Starting upload Aug 7 23:13:54.465726456 (13h34m16.209783322s) [upload=13976]: Done starting upload Aug 7 23:13:54.465831098 (13h34m16.20988796s) [upload=13976]: Uploading segments... Aug 7 23:13:54.466125339 (13h34m16.210182189s) [upload=13976]: Got next segment. Inline: false Aug 7 23:13:54.466207955 (13h34m16.210264835s) [upload=13976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:54.466231453 (13h34m16.21028832s) [upload=13976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:24.194723155 (13h34m45.938780024s) [upload=13976]: Next segment err: context deadline exceeded Aug 7 23:14:24.194812649 (13h34m45.938869515s) [upload=13976]: Done uploading segments... Aug 7 23:14:24.194846807 (13h34m45.938903668s) [upload=13976]: Upload finished. err: context deadline exceeded Aug 7 23:14:24.595327088 (13h34m46.339383935s) [upload=13984]: Starting upload Aug 7 23:14:24.595396123 (13h34m46.339452983s) [upload=13984]: Done starting upload Aug 7 23:14:24.595522981 (13h34m46.339579831s) [upload=13984]: Uploading segments... Aug 7 23:14:24.595654427 (13h34m46.339711274s) [upload=13984]: Got next segment. Inline: false Aug 7 23:14:24.595690189 (13h34m46.339747048s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:24.595695677 (13h34m46.339752537s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:54.326037198 (13h35m16.070094059s) [upload=13984]: Next segment err: context deadline exceeded Aug 7 23:14:54.326107986 (13h35m16.070164835s) [upload=13984]: Done uploading segments... Aug 7 23:14:54.326129117 (13h35m16.07018597s) [upload=13984]: Upload finished. err: context deadline exceeded Aug 7 23:15:12.257450784 (13h35m34.001507635s) [upload=13996]: Starting upload Aug 7 23:15:12.257569128 (13h35m34.00162599s) [upload=13996]: Done starting upload Aug 7 23:15:12.257701999 (13h35m34.001758859s) [upload=13996]: Uploading segments... Aug 7 23:15:12.258022354 (13h35m34.002079211s) [upload=13996]: Got next segment. Inline: false Aug 7 23:15:12.258091150 (13h35m34.002148s) [upload=13996]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:12.258097670 (13h35m34.002154531s) [upload=13996]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:15:41.986853387 (13h36m3.73091025s) [upload=13996]: Next segment err: context deadline exceeded Aug 7 23:15:41.986874552 (13h36m3.730931402s) [upload=13996]: Done uploading segments... Aug 7 23:15:41.986894621 (13h36m3.730951471s) [upload=13996]: Upload finished. err: context deadline exceeded Aug 7 23:16:09.685754413 (13h36m31.429811283s) [upload=14012]: Starting upload Aug 7 23:16:09.685841115 (13h36m31.429897983s) [upload=14012]: Done starting upload Aug 7 23:16:09.686066017 (13h36m31.430122878s) [upload=14012]: Uploading segments... Aug 7 23:16:09.686183581 (13h36m31.430240431s) [upload=14012]: Got next segment. Inline: false Aug 7 23:16:09.686224472 (13h36m31.43028132s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.686231334 (13h36m31.430288193s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:16:39.416007056 (13h37m1.160063926s) [upload=14012]: Next segment err: context deadline exceeded Aug 7 23:16:39.416091190 (13h37m1.160148037s) [upload=14012]: Done uploading segments... Aug 7 23:16:39.416112103 (13h37m1.160168951s) [upload=14012]: Upload finished. err: context deadline exceeded Aug 7 23:17:19.134115876 (13h37m40.87817276s) [upload=14031]: Starting upload Aug 7 23:17:19.134217017 (13h37m40.87827389s) [upload=14031]: Done starting upload Aug 7 23:17:19.134369000 (13h37m40.878425864s) [upload=14031]: Uploading segments... Aug 7 23:17:19.134768595 (13h37m40.878825466s) [upload=14031]: Got next segment. Inline: false Aug 7 23:17:19.134850804 (13h37m40.878907676s) [upload=14031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:19.134856837 (13h37m40.878913708s) [upload=14031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:17:48.861585973 (13h38m10.605642848s) [upload=14031]: Next segment err: context deadline exceeded Aug 7 23:17:48.861771277 (13h38m10.605828148s) [upload=14031]: Done uploading segments... Aug 7 23:17:48.861795148 (13h38m10.605852011s) [upload=14031]: Upload finished. err: context deadline exceeded Duration: 30.000932726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....X............X...............] duration: 3.453604593s FAIL (2025-08-07 23:17:47 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.453638232s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.001638099s FAIL (2025-08-07 23:17:46 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001656533s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................X.XX..X] duration: 30.001239165s FAIL (2025-08-07 23:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001263512s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X.....................................................X.XXX] duration: 1m0.001109275s FAIL (2025-08-07 23:17:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001138995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.02376251s FAIL (2025-08-07 23:17:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.023807265s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XX..XXX] duration: 1m0.001292374s FAIL (2025-08-07 23:17:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (13h28m16.874936092s) [upload=13867]: Starting upload Aug 7 23:07:55.130952807 (13h28m16.875009671s) [upload=13867]: Done starting upload Aug 7 23:07:55.131026070 (13h28m16.87508292s) [upload=13867]: Uploading segments... Aug 7 23:07:55.131220158 (13h28m16.875277027s) [upload=13867]: Got next segment. Inline: true Aug 7 23:07:55.131228194 (13h28m16.875285061s) [upload=13867]: Waiting for error group managing segments... Aug 7 23:07:55.131289715 (13h28m16.875346575s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:55.131295462 (13h28m16.875352322s) [upload=13867]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:55.131300279 (13h28m16.875357151s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:55.281757492 (13h28m17.025814344s) [upload=13867]: Done uploading segments... Aug 7 23:07:55.281776844 (13h28m17.025833694s) [upload=13867]: Upload finished. err: <nil> Aug 7 23:08:53.467366061 (13h29m15.211422912s) [upload=13885]: Starting upload Aug 7 23:08:53.467436784 (13h29m15.211493636s) [upload=13885]: Done starting upload Aug 7 23:08:53.467521314 (13h29m15.211578165s) [upload=13885]: Uploading segments... Aug 7 23:08:53.467570153 (13h29m15.211627016s) [upload=13885]: Got next segment. Inline: true Aug 7 23:08:53.467576092 (13h29m15.211632953s) [upload=13885]: Waiting for error group managing segments... Aug 7 23:08:53.467621884 (13h29m15.211678733s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:53.467627561 (13h29m15.211684422s) [upload=13885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:53.467631218 (13h29m15.211688072s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:53.630342089 (13h29m15.374398952s) [upload=13885]: Done uploading segments... Aug 7 23:08:53.630366467 (13h29m15.37442333s) [upload=13885]: Upload finished. err: <nil> Aug 7 23:10:03.085243036 (13h30m24.829299916s) [upload=13904]: Starting upload Aug 7 23:10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:40.798630406 (13h33m2.542687267s) [upload=13953]: Starting upload Aug 7 23:12:40.798725809 (13h33m2.54278266s) [upload=13953]: Done starting upload Aug 7 23:12:40.798811946 (13h33m2.542868796s) [upload=13953]: Uploading segments... Aug 7 23:12:40.798867069 (13h33m2.542923932s) [upload=13953]: Got next segment. Inline: true Aug 7 23:12:40.798873108 (13h33m2.542929959s) [upload=13953]: Waiting for error group managing segments... Aug 7 23:12:40.798918573 (13h33m2.542975422s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:40.798924603 (13h33m2.542981463s) [upload=13953]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:40.798939855 (13h33m2.542996716s) [upload=13953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.942390421 (13h33m2.686447272s) [upload=13953]: Done uploading segments... Aug 7 23:12:40.942410212 (13h33m2.686467062s) [upload=13953]: Upload finished. err: <nil> Aug 7 23:13:18.137108974 (13h33m39.881165825s) [upload=13967]: Starting upload Aug 7 23:13:18.137180408 (13h33m39.881237292s) [upload=13967]: Done starting upload Aug 7 23:13:18.137258925 (13h33m39.881315786s) [upload=13967]: Uploading segments... Aug 7 23:13:18.137309635 (13h33m39.881366505s) [upload=13967]: Got next segment. Inline: true Aug 7 23:13:18.137314511 (13h33m39.881371371s) [upload=13967]: Waiting for error group managing segments... Aug 7 23:13:18.137374772 (13h33m39.881431633s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:18.137380412 (13h33m39.881437273s) [upload=13967]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:18.137383733 (13h33m39.881440593s) [upload=13967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:18.301480519 (13h33m40.045537379s) [upload=13967]: Done uploading segments... Aug 7 23:13:18.301498918 (13h33m40.045555767s) [upload=13967]: Upload finished. err: <nil> Aug 7 23:14:13.169029066 (13h34m34.913085913s) [upload=13980]: Starting upload Aug 7 23:14:13.169095254 (13h34m34.913152101s) [upload=13980]: Done starting upload Aug 7 23:14:13.169153338 (13h34m34.913210184s) [upload=13980]: Uploading segments... Aug 7 23:14:13.169225662 (13h34m34.913282508s) [upload=13980]: Got next segment. Inline: true Aug 7 23:14:13.169230885 (13h34m34.913287744s) [upload=13980]: Waiting for error group managing segments... Aug 7 23:14:13.169272521 (13h34m34.913329381s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:13.169288952 (13h34m34.913345813s) [upload=13980]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:13.169291623 (13h34m34.913348483s) [upload=13980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:12.895960606 (13h35m34.640017467s) [upload=13980]: Done uploading segments... Aug 7 23:15:12.895983608 (13h35m34.640040456s) [upload=13980]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:13.308146067 (13h35m35.052202918s) [upload=13997]: Starting upload Aug 7 23:15:13.308219271 (13h35m35.052276134s) [upload=13997]: Done starting upload Aug 7 23:15:13.308297319 (13h35m35.052354171s) [upload=13997]: Uploading segments... Aug 7 23:15:13.308360203 (13h35m35.052417072s) [upload=13997]: Got next segment. Inline: true Aug 7 23:15:13.308365548 (13h35m35.052422408s) [upload=13997]: Waiting for error group managing segments... Aug 7 23:15:13.308409794 (13h35m35.052466657s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:13.308426102 (13h35m35.052482963s) [upload=13997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:13.308429725 (13h35m35.052486585s) [upload=13997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:13.040589970 (13h36m34.784646843s) [upload=13997]: Done uploading segments... Aug 7 23:16:13.040614428 (13h36m34.784671274s) [upload=13997]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:20.767866726 (13h36m42.511923577s) [upload=14018]: Starting upload Aug 7 23:16:20.767927544 (13h36m42.511984398s) [upload=14018]: Done starting upload Aug 7 23:16:20.767993090 (13h36m42.512049942s) [upload=14018]: Uploading segments... Aug 7 23:16:20.768055054 (13h36m42.512111926s) [upload=14018]: Got next segment. Inline: true Aug 7 23:16:20.768061500 (13h36m42.512118351s) [upload=14018]: Waiting for error group managing segments... Aug 7 23:16:20.768130604 (13h36m42.512187577s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:20.768139327 (13h36m42.512196192s) [upload=14018]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:20.768144987 (13h36m42.512201849s) [upload=14018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:20.496845379 (13h37m42.240902231s) [upload=14018]: Done uploading segments... Aug 7 23:17:20.496873778 (13h37m42.240930632s) [upload=14018]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001321881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXX.X] duration: 30.001470305s FAIL (2025-08-07 23:17:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001511207s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.000869896s FAIL (2025-08-07 23:17:16 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000901748s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 1.014335287s FAIL (2025-08-07 23:17:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.014350716s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 2m0.000882389s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:17:05 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.000912481s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X.XX..] duration: 2.476705854s FAIL (2025-08-07 23:16:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.476727393s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X............X..............] duration: 3.085144097s FAIL (2025-08-07 23:16:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.085175418s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXX.XXXX] duration: 30.001432412s FAIL (2025-08-07 23:16:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 756435112s) [upload=13913]: Done starting upload Aug 7 23:10:28.012443618 (13h30m49.756500468s) [upload=13913]: Uploading segments... Aug 7 23:10:28.012512066 (13h30m49.756568934s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:28.012518615 (13h30m49.756575478s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:28.012569185 (13h30m49.756626036s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:28.012575496 (13h30m49.756632345s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:28.012578877 (13h30m49.756635741s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:28.178423527 (13h30m49.922480378s) [upload=13913]: Done uploading segments... Aug 7 23:10:28.178443696 (13h30m49.922500546s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:16.095264069 (13h31m37.839320941s) [upload=13924]: Starting upload Aug 7 23:11:16.095345689 (13h31m37.83940255s) [upload=13924]: Done starting upload Aug 7 23:11:16.095421525 (13h31m37.839478385s) [upload=13924]: Uploading segments... Aug 7 23:11:16.095460332 (13h31m37.839517201s) [upload=13924]: Got next segment. Inline: true Aug 7 23:11:16.095465792 (13h31m37.839522653s) [upload=13924]: Waiting for error group managing segments... Aug 7 23:11:16.095512445 (13h31m37.839569308s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:16.095517610 (13h31m37.839574471s) [upload=13924]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:16.095521136 (13h31m37.839577996s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:16.231539768 (13h31m37.975596629s) [upload=13924]: Done uploading segments... Aug 7 23:11:16.231558376 (13h31m37.975615223s) [upload=13924]: Upload finished. err: <nil> Aug 7 23:11:48.329873818 (13h32m10.073930665s) [upload=13936]: Starting upload Aug 7 23:11:48.329969166 (13h32m10.074026026s) [upload=13936]: Done starting upload Aug 7 23:11:48.330046481 (13h32m10.074103328s) [upload=13936]: Uploading segments... Aug 7 23:11:48.330100337 (13h32m10.074157203s) [upload=13936]: Got next segment. Inline: true Aug 7 23:11:48.330105628 (13h32m10.074162488s) [upload=13936]: Waiting for error group managing segments... Aug 7 23:11:48.330144233 (13h32m10.074201079s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.330148601 (13h32m10.074205447s) [upload=13936]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.330151121 (13h32m10.074207967s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:18.060862417 (13h32m39.804919268s) [upload=13936]: Done uploading segments... Aug 7 23:12:18.060910946 (13h32m39.804967806s) [upload=13936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:19.230822944 (13h32m40.974879804s) [upload=13947]: Starting upload Aug 7 23:12:19.230891930 (13h32m40.97494879s) [upload=13947]: Done starting upload Aug 7 23:12:19.230961121 (13h32m40.975017968s) [upload=13947]: Uploading segments... Aug 7 23:12:19.231005268 (13h32m40.975062139s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:19.231009941 (13h32m40.975066801s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:19.231058939 (13h32m40.975115785s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:19.231063293 (13h32m40.975120154s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:19.231065710 (13h32m40.97512257s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:48.962967239 (13h33m10.707024101s) [upload=13947]: Done uploading segments... Aug 7 23:12:48.962986733 (13h33m10.707043592s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:03.240322668 (13h33m24.984379539s) [upload=13961]: Starting upload Aug 7 23:13:03.240467142 (13h33m24.984524008s) [upload=13961]: Done starting upload Aug 7 23:13:03.240603062 (13h33m24.984659911s) [upload=13961]: Uploading segments... Aug 7 23:13:03.240714255 (13h33m24.984771129s) [upload=13961]: Got next segment. Inline: true Aug 7 23:13:03.240723064 (13h33m24.984779933s) [upload=13961]: Waiting for error group managing segments... Aug 7 23:13:03.240804558 (13h33m24.984861428s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:03.240811567 (13h33m24.984868436s) [upload=13961]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:03.240816509 (13h33m24.984873376s) [upload=13961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:32.970033245 (13h33m54.714090121s) [upload=13961]: Done uploading segments... Aug 7 23:13:32.970064842 (13h33m54.714121693s) [upload=13961]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:33.400576508 (13h33m55.144633367s) [upload=13970]: Starting upload Aug 7 23:13:33.400642971 (13h33m55.144699817s) [upload=13970]: Done starting upload Aug 7 23:13:33.400723341 (13h33m55.144780187s) [upload=13970]: Uploading segments... Aug 7 23:13:33.400775896 (13h33m55.144832764s) [upload=13970]: Got next segment. Inline: true Aug 7 23:13:33.400780746 (13h33m55.144837592s) [upload=13970]: Waiting for error group managing segments... Aug 7 23:13:33.400844129 (13h33m55.144900979s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:33.400851223 (13h33m55.144908092s) [upload=13970]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:33.400854888 (13h33m55.144911755s) [upload=13970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:33.576404182 (13h33m55.320461054s) [upload=13970]: Done uploading segments... Aug 7 23:13:33.576428692 (13h33m55.320485559s) [upload=13970]: Upload finished. err: <nil> Aug 7 23:13:55.968242604 (13h34m17.712299464s) [upload=13977]: Starting upload Aug 7 23:13:55.968304238 (13h34m17.712361108s) [upload=13977]: Done starting upload Aug 7 23:13:55.968379243 (13h34m17.712436104s) [upload=13977]: Uploading segments... Aug 7 23:13:55.968458505 (13h34m17.712515376s) [upload=13977]: Got next segment. Inline: true Aug 7 23:13:55.968466277 (13h34m17.71252314s) [upload=13977]: Waiting for error group managing segments... Aug 7 23:13:55.968540306 (13h34m17.712597182s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:55.968547187 (13h34m17.712604053s) [upload=13977]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:55.968551707 (13h34m17.712608571s) [upload=13977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:25.692983305 (13h34m47.437040166s) [upload=13977]: Done uploading segments... Aug 7 23:14:25.693002238 (13h34m47.437059088s) [upload=13977]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:37.691964145 (13h34m59.436021005s) [upload=13988]: Starting upload Aug 7 23:14:37.692037111 (13h34m59.436093974s) [upload=13988]: Done starting upload Aug 7 23:14:37.692096199 (13h34m59.436153049s) [upload=13988]: Uploading segments... Aug 7 23:14:37.692153333 (13h34m59.436210258s) [upload=13988]: Got next segment. Inline: true Aug 7 23:14:37.692158481 (13h34m59.436215342s) [upload=13988]: Waiting for error group managing segments... Aug 7 23:14:37.692215139 (13h34m59.436271999s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:37.692221507 (13h34m59.436278368s) [upload=13988]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:37.692226533 (13h34m59.436283398s) [upload=13988]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:07.421874116 (13h35m29.165930978s) [upload=13988]: Done uploading segments... Aug 7 23:15:07.421902858 (13h35m29.165959703s) [upload=13988]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:07.832637004 (13h35m29.576693853s) [upload=13995]: Starting upload Aug 7 23:15:07.832719564 (13h35m29.576776416s) [upload=13995]: Done starting upload Aug 7 23:15:07.832826894 (13h35m29.576883745s) [upload=13995]: Uploading segments... Aug 7 23:15:07.832887307 (13h35m29.576944167s) [upload=13995]: Got next segment. Inline: true Aug 7 23:15:07.832893154 (13h35m29.576950015s) [upload=13995]: Waiting for error group managing segments... Aug 7 23:15:07.832938815 (13h35m29.576995666s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:07.832943939 (13h35m29.5770008s) [upload=13995]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:07.832947902 (13h35m29.577004762s) [upload=13995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:37.560683179 (13h35m59.304740027s) [upload=13995]: Done uploading segments... Aug 7 23:15:37.560704530 (13h35m59.304761377s) [upload=13995]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:13.030124119 (13h36m34.774180979s) [upload=14013]: Starting upload Aug 7 23:16:13.030198850 (13h36m34.774255715s) [upload=14013]: Done starting upload Aug 7 23:16:13.030287975 (13h36m34.774344821s) [upload=14013]: Uploading segments... Aug 7 23:16:13.030344552 (13h36m34.774401422s) [upload=14013]: Got next segment. Inline: true Aug 7 23:16:13.030349665 (13h36m34.774406525s) [upload=14013]: Waiting for error group managing segments... Aug 7 23:16:13.030401514 (13h36m34.774458375s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:13.030406613 (13h36m34.774463474s) [upload=14013]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:13.030409548 (13h36m34.774466408s) [upload=14013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:42.761381352 (13h37m4.505438199s) [upload=14013]: Done uploading segments... Aug 7 23:16:42.761401933 (13h37m4.505458793s) [upload=14013]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00145437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 857.683642ms FAIL (2025-08-07 23:16:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 857.781807ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 30.001291079s FAIL (2025-08-07 23:16:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001319867s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)