Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XXXX.X.....................................................X] duration: 1m0.000434804s FAIL (2025-08-07 23:13:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000529227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...............................X....X............X........] duration: 3.084033939s FAIL (2025-08-07 23:12:58 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.084062942s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 914.988039ms FAIL (2025-08-07 23:12:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 915.023826ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXX..X.....................................................X] duration: 30.000648379s FAIL (2025-08-07 23:12:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== load=13939]: Done starting upload Aug 7 23:12:06.709058291 (13h32m28.453115142s) [upload=13939]: Uploading segments... Aug 7 23:12:06.709414773 (13h32m28.453471633s) [upload=13939]: Got next segment. Inline: false Aug 7 23:12:06.709447870 (13h32m28.45350473s) [upload=13939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.709453310 (13h32m28.453510171s) [upload=13939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:12:06.820298464 (13h32m28.564355315s) [seg_pos={0 0}, upload=13939]: Begin upload segment... Aug 7 23:12:06.820353704 (13h32m28.564410564s) [seg_pos={0 0}, upload=13939]: Done begin upload segment. Aug 7 23:12:06.820390101 (13h32m28.564446951s) [noise=true , address=104.36.82.8:4969 , piece_id=7MQR2SVUM76G22K6RUTCJKMERMX4HLX4EAMZJMZS6D3UVYYHYCVQ, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.820448965 (13h32m28.564505831s) [noise=true , address=77.90.31.21:4878 , piece_id=IO7M6EL3TAE6WPD6WNKQMU7JYHY4OL3E4E46JNEXVKQQGSWDMNSQ, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.820815881 (13h32m28.564872743s) [noise=true , address=77.90.31.19:4841 , piece_id=BCXQUB2X5VVIFCHUAHQGD43UAMNTNBPO2ZMZ6LF4TPZBMBER2NJA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.820893015 (13h32m28.56494988s) [noise=true , address=77.90.31.17:4772 , piece_id=JHDYI7MUFLJRWVH3IUOIKHAIAUPNIQNDTRTWOIBH7QU4T5NH6RZQ, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.821141553 (13h32m28.565198416s) [noise=true , address=77.90.31.14:4661 , piece_id=HWELIBFLJC6K2IPVV2J5U7O55IKOBUOLLHLMVBFG6AVIEXMP24LA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.821280429 (13h32m28.565337301s) [noise=true , address=104.36.82.6:4876 , piece_id=Y2HFXOS6O7DOGA57LYIQCTHG46PF5UILJGF2VJRJ2XXYUZBGLSCA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.821459086 (13h32m28.565515949s) [noise=true , address=104.36.82.2:4730 , piece_id=GZPRAOONQUIG43X5ODD2F22NNMQWCOYUIU66LL6SWJWVWTZ6R36Q, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.821718550 (13h32m28.565775416s) [noise=true , address=45.91.116.7:4010 , piece_id=KGKZWJIJRCKP2KAM2BXH5EQXZU6AY4SKU3YIJTUIMPEJMHBSBTXA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.821753161 (13h32m28.565810021s) [noise=true , address=45.91.116.4:4939 , piece_id=X3TBWW55NKZJEZBLZYONWHUHEZ5ROJELSKUYSQNH277IMZRSLDNA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.822141842 (13h32m28.566198711s) [noise=true , address=104.36.82.10:4078 , piece_id=2JLXDRSXGXI5AHV7QGNZUDPKPS2IMZLOSINYBYCXF7YFFXXTP5PA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.822175057 (13h32m28.56623192s) [noise=true , address=45.91.116.9:4261 , piece_id=YQSAMMSUMO6D6FISYIXASKMTXBUNN3HTICAJSI5MIQCAXC72TY7A, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.822526653 (13h32m28.56658352s) [noise=true , address=104.36.82.3:4767 , piece_id=LOHVDMJ6KZPMJZPDRAHT3URYE4EKVITYSG7VVVFIBFCZE6NQKHYA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.822588019 (13h32m28.566644883s) [noise=true , address=104.36.82.16:4293 , piece_id=UT5KWEGACHBACWZOJGFYLOYU7PFDE654GAQX6H26QZXUCPBLC5ZA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.822893699 (13h32m28.566950559s) [noise=true , address=45.91.116.13:4154 , piece_id=OYQRDW6EPXAONOYTIGW4HQTXBUH53VSM4IVIYMKQOPNMGPOHOR2Q, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.822976231 (13h32m28.567033095s) [noise=true , address=104.36.82.15:4258 , piece_id=LGDTVWJPTQOQ6UEYQC3ROTQXOKJ4QOSA2J5NM6WPYAATSEUTIF5Q, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:06.823520823 (13h32m28.567577687s) [noise=true , address=77.90.31.11:4554 , piece_id=PMGZKAJ2VRGR5H3KQBKL5SHCGSU5DRDGEZBNPKI7KFNO5AHHNKCA, seg_pos={0 0}, upload=13939]: Uploading piece... Aug 7 23:12:07.469968585 (13h32m29.214025446s) [upload=13939]: Next returned nil segment Aug 7 23:12:07.469988878 (13h32m29.214045749s) [upload=13939]: Waiting for error group managing segments... Aug 7 23:12:07.637018934 (13h32m29.381075785s) [noise=true , address=104.36.82.16:4293 , piece_id=UT5KWEGACHBACWZOJGFYLOYU7PFDE654GAQX6H26QZXUCPBLC5ZA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.642079304 (13h32m29.38613615s) [noise=true , address=104.36.82.15:4258 , piece_id=LGDTVWJPTQOQ6UEYQC3ROTQXOKJ4QOSA2J5NM6WPYAATSEUTIF5Q, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.642281639 (13h32m29.386338486s) [noise=true , address=104.36.82.10:4078 , piece_id=2JLXDRSXGXI5AHV7QGNZUDPKPS2IMZLOSINYBYCXF7YFFXXTP5PA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.650188428 (13h32m29.39424528s) [noise=true , address=104.36.82.3:4767 , piece_id=LOHVDMJ6KZPMJZPDRAHT3URYE4EKVITYSG7VVVFIBFCZE6NQKHYA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.650518412 (13h32m29.394575271s) [noise=true , address=45.91.116.4:4939 , piece_id=X3TBWW55NKZJEZBLZYONWHUHEZ5ROJELSKUYSQNH277IMZRSLDNA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.678763731 (13h32m29.422820591s) [noise=true , address=77.90.31.21:4878 , piece_id=IO7M6EL3TAE6WPD6WNKQMU7JYHY4OL3E4E46JNEXVKQQGSWDMNSQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.691257838 (13h32m29.435314704s) [noise=true , address=45.91.116.7:4010 , piece_id=KGKZWJIJRCKP2KAM2BXH5EQXZU6AY4SKU3YIJTUIMPEJMHBSBTXA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.698168459 (13h32m29.442225309s) [noise=true , address=45.91.116.9:4261 , piece_id=YQSAMMSUMO6D6FISYIXASKMTXBUNN3HTICAJSI5MIQCAXC72TY7A, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.708048205 (13h32m29.452105056s) [noise=true , address=104.36.82.6:4876 , piece_id=Y2HFXOS6O7DOGA57LYIQCTHG46PF5UILJGF2VJRJ2XXYUZBGLSCA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.724323502 (13h32m29.468380349s) [noise=true , address=77.90.31.14:4661 , piece_id=HWELIBFLJC6K2IPVV2J5U7O55IKOBUOLLHLMVBFG6AVIEXMP24LA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.788084295 (13h32m29.532141146s) [noise=true , address=77.90.31.11:4554 , piece_id=PMGZKAJ2VRGR5H3KQBKL5SHCGSU5DRDGEZBNPKI7KFNO5AHHNKCA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.790692188 (13h32m29.534749035s) [noise=true , address=77.90.31.19:4841 , piece_id=BCXQUB2X5VVIFCHUAHQGD43UAMNTNBPO2ZMZ6LF4TPZBMBER2NJA, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.827285350 (13h32m29.571342201s) [noise=true , address=77.90.31.17:4772 , piece_id=JHDYI7MUFLJRWVH3IUOIKHAIAUPNIQNDTRTWOIBH7QU4T5NH6RZQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: <nil> Aug 7 23:12:07.827336330 (13h32m29.57139319s) [seg_pos={0 0}, upload=13939]: Segment reached optimal threshold of 13 pieces. Aug 7 23:12:07.827615018 (13h32m29.571671881s) [noise=true , address=45.91.116.13:4154 , piece_id=OYQRDW6EPXAONOYTIGW4HQTXBUH53VSM4IVIYMKQOPNMGPOHOR2Q, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 23:12:07.827796613 (13h32m29.571853464s) [noise=true , address=104.36.82.8:4969 , piece_id=7MQR2SVUM76G22K6RUTCJKMERMX4HLX4EAMZJMZS6D3UVYYHYCVQ, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38538->104.36.82.8:4969: use of closed network connection Aug 7 23:12:07.827901392 (13h32m29.571958243s) [noise=true , address=104.36.82.2:4730 , piece_id=GZPRAOONQUIG43X5ODD2F22NNMQWCOYUIU66LL6SWJWVWTZ6R36Q, seg_pos={0 0}, upload=13939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 139.59.144.22:60032->104.36.82.2:4730: use of closed network connection; context canceled Aug 7 23:12:07.827937880 (13h32m29.571994732s) [seg_pos={0 0}, upload=13939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:12:07.828022548 (13h32m29.572079408s) [upload=13939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:12:07.828027828 (13h32m29.572084689s) [upload=13939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:07.965146491 (13h32m29.709203382s) [upload=13939]: Done uploading segments... Aug 7 23:12:07.965172031 (13h32m29.709228879s) [upload=13939]: Upload finished. err: <nil> Aug 7 23:12:24.531927541 (13h32m46.275984394s) [upload=13949]: Starting upload Aug 7 23:12:24.532011658 (13h32m46.276068518s) [upload=13949]: Done starting upload Aug 7 23:12:24.532148550 (13h32m46.276205412s) [upload=13949]: Uploading segments... Aug 7 23:12:24.532297280 (13h32m46.27635414s) [upload=13949]: Got next segment. Inline: false Aug 7 23:12:24.532373131 (13h32m46.27643s) [upload=13949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:24.532381517 (13h32m46.276438387s) [upload=13949]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:12:54.256553034 (13h33m16.000609899s) [upload=13949]: Next segment err: context deadline exceeded Aug 7 23:12:54.256729208 (13h33m16.000786059s) [upload=13949]: Done uploading segments... Aug 7 23:12:54.256746630 (13h33m16.000803484s) [upload=13949]: Upload finished. err: context deadline exceeded Duration: 30.000676408s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.041309613s FAIL (2025-08-07 23:12:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.041352029s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X.......................................................XX] duration: 30.001092011s FAIL (2025-08-07 23:12:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (13h25m27.358331571s) [upload=13819]: Upload finished. err: <nil> Aug 7 23:05:37.597719431 (13h25m59.341776283s) [upload=13827]: Starting upload Aug 7 23:05:37.597786920 (13h25m59.341843774s) [upload=13827]: Done starting upload Aug 7 23:05:37.597872453 (13h25m59.341929305s) [upload=13827]: Uploading segments... Aug 7 23:05:37.597954792 (13h25m59.342011657s) [upload=13827]: Got next segment. Inline: true Aug 7 23:05:37.597960599 (13h25m59.342017459s) [upload=13827]: Waiting for error group managing segments... Aug 7 23:05:37.598010394 (13h25m59.342067243s) [upload=13827]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:05:37.598015744 (13h25m59.342072605s) [upload=13827]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:37.598019424 (13h25m59.342076283s) [upload=13827]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:37.749090862 (13h25m59.493147712s) [upload=13827]: Done uploading segments... Aug 7 23:05:37.749110397 (13h25m59.493167247s) [upload=13827]: Upload finished. err: <nil> Aug 7 23:06:41.828111526 (13h27m3.572168391s) [upload=13842]: Starting upload Aug 7 23:06:41.828216224 (13h27m3.572273076s) [upload=13842]: Done starting upload Aug 7 23:06:41.828334659 (13h27m3.572391512s) [upload=13842]: Uploading segments... Aug 7 23:06:41.828449287 (13h27m3.572506151s) [upload=13842]: Got next segment. Inline: true Aug 7 23:06:41.828457932 (13h27m3.572514797s) [upload=13842]: Waiting for error group managing segments... Aug 7 23:06:41.828522114 (13h27m3.572578976s) [upload=13842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:41.828529109 (13h27m3.57258597s) [upload=13842]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:41.828532670 (13h27m3.572589532s) [upload=13842]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:41.979430299 (13h27m3.72348716s) [upload=13842]: Done uploading segments... Aug 7 23:06:41.979453508 (13h27m3.723510358s) [upload=13842]: Upload finished. err: <nil> Aug 7 23:07:32.461706064 (13h27m54.205762928s) [upload=13858]: Starting upload Aug 7 23:07:32.461824536 (13h27m54.205881399s) [upload=13858]: Done starting upload Aug 7 23:07:32.461961765 (13h27m54.206018623s) [upload=13858]: Uploading segments... Aug 7 23:07:32.462039185 (13h27m54.206096056s) [upload=13858]: Got next segment. Inline: true Aug 7 23:07:32.462045894 (13h27m54.206102758s) [upload=13858]: Waiting for error group managing segments... Aug 7 23:07:32.462110153 (13h27m54.206167013s) [upload=13858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:32.462116381 (13h27m54.206173242s) [upload=13858]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:32.462119766 (13h27m54.206176626s) [upload=13858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:32.615220579 (13h27m54.359277439s) [upload=13858]: Done uploading segments... Aug 7 23:07:32.615246418 (13h27m54.359303279s) [upload=13858]: Upload finished. err: <nil> Aug 7 23:08:40.493251780 (13h29m2.237308632s) [upload=13880]: Starting upload Aug 7 23:08:40.493307346 (13h29m2.237364211s) [upload=13880]: Done starting upload Aug 7 23:08:40.493377552 (13h29m2.237434398s) [upload=13880]: Uploading segments... Aug 7 23:08:40.493449296 (13h29m2.237506164s) [upload=13880]: Got next segment. Inline: true Aug 7 23:08:40.493457374 (13h29m2.237514224s) [upload=13880]: Waiting for error group managing segments... Aug 7 23:08:40.493501643 (13h29m2.237558494s) [upload=13880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:40.493507544 (13h29m2.237564404s) [upload=13880]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:40.493510790 (13h29m2.23756764s) [upload=13880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:40.636364852 (13h29m2.380421716s) [upload=13880]: Done uploading segments... Aug 7 23:08:40.636386281 (13h29m2.38044313s) [upload=13880]: Upload finished. err: <nil> Aug 7 23:09:01.297603828 (13h29m23.041660691s) [upload=13886]: Starting upload Aug 7 23:09:01.297709608 (13h29m23.04176648s) [upload=13886]: Done starting upload Aug 7 23:09:01.297804098 (13h29m23.041860954s) [upload=13886]: Uploading segments... Aug 7 23:09:01.297882202 (13h29m23.041939076s) [upload=13886]: Got next segment. Inline: true Aug 7 23:09:01.297889366 (13h29m23.041946233s) [upload=13886]: Waiting for error group managing segments... Aug 7 23:09:01.297998616 (13h29m23.04205548s) [upload=13886]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:01.298006016 (13h29m23.042062881s) [upload=13886]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:01.298010262 (13h29m23.042067126s) [upload=13886]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:01.452127466 (13h29m23.196184318s) [upload=13886]: Done uploading segments... Aug 7 23:09:01.452173116 (13h29m23.196229988s) [upload=13886]: Upload finished. err: <nil> Aug 7 23:09:27.855960951 (13h29m49.600017802s) [upload=13891]: Starting upload Aug 7 23:09:27.856047975 (13h29m49.600104835s) [upload=13891]: Done starting upload Aug 7 23:09:27.856128881 (13h29m49.600185732s) [upload=13891]: Uploading segments... Aug 7 23:09:27.856217016 (13h29m49.600273883s) [upload=13891]: Got next segment. Inline: true Aug 7 23:09:27.856223587 (13h29m49.600280447s) [upload=13891]: Waiting for error group managing segments... Aug 7 23:09:27.856283213 (13h29m49.600340073s) [upload=13891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:27.856289411 (13h29m49.600346272s) [upload=13891]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:27.856292931 (13h29m49.600349791s) [upload=13891]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:28.011857390 (13h29m49.75591424s) [upload=13891]: Done uploading segments... Aug 7 23:09:28.011879735 (13h29m49.755936584s) [upload=13891]: Upload finished. err: <nil> Aug 7 23:10:28.012300991 (13h30m49.756357841s) [upload=13913]: Starting upload Aug 7 23:10:28.012378252 (13h30m49.756435112s) [upload=13913]: Done starting upload Aug 7 23:10:28.012443618 (13h30m49.756500468s) [upload=13913]: Uploading segments... Aug 7 23:10:28.012512066 (13h30m49.756568934s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:28.012518615 (13h30m49.756575478s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:28.012569185 (13h30m49.756626036s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:28.012575496 (13h30m49.756632345s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:28.012578877 (13h30m49.756635741s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:28.178423527 (13h30m49.922480378s) [upload=13913]: Done uploading segments... Aug 7 23:10:28.178443696 (13h30m49.922500546s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:16.095264069 (13h31m37.839320941s) [upload=13924]: Starting upload Aug 7 23:11:16.095345689 (13h31m37.83940255s) [upload=13924]: Done starting upload Aug 7 23:11:16.095421525 (13h31m37.839478385s) [upload=13924]: Uploading segments... Aug 7 23:11:16.095460332 (13h31m37.839517201s) [upload=13924]: Got next segment. Inline: true Aug 7 23:11:16.095465792 (13h31m37.839522653s) [upload=13924]: Waiting for error group managing segments... Aug 7 23:11:16.095512445 (13h31m37.839569308s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:16.095517610 (13h31m37.839574471s) [upload=13924]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:16.095521136 (13h31m37.839577996s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:16.231539768 (13h31m37.975596629s) [upload=13924]: Done uploading segments... Aug 7 23:11:16.231558376 (13h31m37.975615223s) [upload=13924]: Upload finished. err: <nil> Aug 7 23:11:48.329873818 (13h32m10.073930665s) [upload=13936]: Starting upload Aug 7 23:11:48.329969166 (13h32m10.074026026s) [upload=13936]: Done starting upload Aug 7 23:11:48.330046481 (13h32m10.074103328s) [upload=13936]: Uploading segments... Aug 7 23:11:48.330100337 (13h32m10.074157203s) [upload=13936]: Got next segment. Inline: true Aug 7 23:11:48.330105628 (13h32m10.074162488s) [upload=13936]: Waiting for error group managing segments... Aug 7 23:11:48.330144233 (13h32m10.074201079s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.330148601 (13h32m10.074205447s) [upload=13936]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.330151121 (13h32m10.074207967s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:18.060862417 (13h32m39.804919268s) [upload=13936]: Done uploading segments... Aug 7 23:12:18.060910946 (13h32m39.804967806s) [upload=13936]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:19.230822944 (13h32m40.974879804s) [upload=13947]: Starting upload Aug 7 23:12:19.230891930 (13h32m40.97494879s) [upload=13947]: Done starting upload Aug 7 23:12:19.230961121 (13h32m40.975017968s) [upload=13947]: Uploading segments... Aug 7 23:12:19.231005268 (13h32m40.975062139s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:19.231009941 (13h32m40.975066801s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:19.231058939 (13h32m40.975115785s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:19.231063293 (13h32m40.975120154s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:19.231065710 (13h32m40.97512257s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:48.962967239 (13h33m10.707024101s) [upload=13947]: Done uploading segments... Aug 7 23:12:48.962986733 (13h33m10.707043592s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001116533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [XX.......................................................XX.] duration: 874.171664ms FAIL (2025-08-07 23:12:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5.087944464 (13h22m16.832001314s) [upload=13761]: Upload finished. err: <nil> Aug 7 23:02:35.452281267 (13h22m57.196338128s) [upload=13772]: Starting upload Aug 7 23:02:35.452363022 (13h22m57.196419874s) [upload=13772]: Done starting upload Aug 7 23:02:35.452463263 (13h22m57.196520113s) [upload=13772]: Uploading segments... Aug 7 23:02:35.452523098 (13h22m57.196579966s) [upload=13772]: Got next segment. Inline: true Aug 7 23:02:35.452528530 (13h22m57.19658539s) [upload=13772]: Waiting for error group managing segments... Aug 7 23:02:35.452574263 (13h22m57.196631123s) [upload=13772]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:02:35.452579819 (13h22m57.196636679s) [upload=13772]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:02:35.452583343 (13h22m57.196640204s) [upload=13772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:02:35.594791272 (13h22m57.338848123s) [upload=13772]: Done uploading segments... Aug 7 23:02:35.594825533 (13h22m57.338882393s) [upload=13772]: Upload finished. err: <nil> Aug 7 23:03:49.636350992 (13h24m11.380407843s) [upload=13795]: Starting upload Aug 7 23:03:49.636417823 (13h24m11.380474677s) [upload=13795]: Done starting upload Aug 7 23:03:49.636490213 (13h24m11.380547076s) [upload=13795]: Uploading segments... Aug 7 23:03:49.636567311 (13h24m11.380624185s) [upload=13795]: Got next segment. Inline: true Aug 7 23:03:49.636574436 (13h24m11.380631287s) [upload=13795]: Waiting for error group managing segments... Aug 7 23:03:49.636645517 (13h24m11.380702374s) [upload=13795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:03:49.636655113 (13h24m11.380711973s) [upload=13795]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:03:49.636658487 (13h24m11.380715343s) [upload=13795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:03:49.781691324 (13h24m11.525748175s) [upload=13795]: Done uploading segments... Aug 7 23:03:49.781713904 (13h24m11.52577076s) [upload=13795]: Upload finished. err: <nil> Aug 7 23:04:57.312274559 (13h25m19.05633143s) [upload=13814]: Starting upload Aug 7 23:04:57.312366372 (13h25m19.056423226s) [upload=13814]: Done starting upload Aug 7 23:04:57.312497707 (13h25m19.056554555s) [upload=13814]: Uploading segments... Aug 7 23:04:57.312567762 (13h25m19.05662463s) [upload=13814]: Got next segment. Inline: true Aug 7 23:04:57.312574196 (13h25m19.056631067s) [upload=13814]: Waiting for error group managing segments... Aug 7 23:04:57.312653597 (13h25m19.056710478s) [upload=13814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:04:57.312660409 (13h25m19.056717278s) [upload=13814]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:04:57.312664013 (13h25m19.056720884s) [upload=13814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:04:57.459986021 (13h25m19.20404287s) [upload=13814]: Done uploading segments... Aug 7 23:04:57.460005087 (13h25m19.204061937s) [upload=13814]: Upload finished. err: <nil> Aug 7 23:05:47.352420719 (13h26m9.096477638s) [upload=13831]: Starting upload Aug 7 23:05:47.352495951 (13h26m9.096552804s) [upload=13831]: Done starting upload Aug 7 23:05:47.352578458 (13h26m9.096635308s) [upload=13831]: Uploading segments... Aug 7 23:05:47.352621273 (13h26m9.096678143s) [upload=13831]: Got next segment. Inline: true Aug 7 23:05:47.352626460 (13h26m9.09668331s) [upload=13831]: Waiting for error group managing segments... Aug 7 23:05:47.352677569 (13h26m9.096734418s) [upload=13831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:05:47.352682758 (13h26m9.096739608s) [upload=13831]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:47.352686503 (13h26m9.096743353s) [upload=13831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:47.525022266 (13h26m9.26907913s) [upload=13831]: Done uploading segments... Aug 7 23:05:47.525047633 (13h26m9.269104482s) [upload=13831]: Upload finished. err: <nil> Aug 7 23:06:58.801209002 (13h27m20.545265866s) [upload=13850]: Starting upload Aug 7 23:06:58.801309642 (13h27m20.545366507s) [upload=13850]: Done starting upload Aug 7 23:06:58.801447578 (13h27m20.545504443s) [upload=13850]: Uploading segments... Aug 7 23:06:58.801538129 (13h27m20.545594996s) [upload=13850]: Got next segment. Inline: true Aug 7 23:06:58.801545498 (13h27m20.54560237s) [upload=13850]: Waiting for error group managing segments... Aug 7 23:06:58.801617944 (13h27m20.545674808s) [upload=13850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:58.801623960 (13h27m20.545680824s) [upload=13850]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:58.801630296 (13h27m20.545687164s) [upload=13850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:58.963809105 (13h27m20.707865989s) [upload=13850]: Done uploading segments... Aug 7 23:06:58.963847876 (13h27m20.707904747s) [upload=13850]: Upload finished. err: <nil> Aug 7 23:07:55.130879231 (13h28m16.874936092s) [upload=13867]: Starting upload Aug 7 23:07:55.130952807 (13h28m16.875009671s) [upload=13867]: Done starting upload Aug 7 23:07:55.131026070 (13h28m16.87508292s) [upload=13867]: Uploading segments... Aug 7 23:07:55.131220158 (13h28m16.875277027s) [upload=13867]: Got next segment. Inline: true Aug 7 23:07:55.131228194 (13h28m16.875285061s) [upload=13867]: Waiting for error group managing segments... Aug 7 23:07:55.131289715 (13h28m16.875346575s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:55.131295462 (13h28m16.875352322s) [upload=13867]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:55.131300279 (13h28m16.875357151s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:55.281757492 (13h28m17.025814344s) [upload=13867]: Done uploading segments... Aug 7 23:07:55.281776844 (13h28m17.025833694s) [upload=13867]: Upload finished. err: <nil> Aug 7 23:08:53.467366061 (13h29m15.211422912s) [upload=13885]: Starting upload Aug 7 23:08:53.467436784 (13h29m15.211493636s) [upload=13885]: Done starting upload Aug 7 23:08:53.467521314 (13h29m15.211578165s) [upload=13885]: Uploading segments... Aug 7 23:08:53.467570153 (13h29m15.211627016s) [upload=13885]: Got next segment. Inline: true Aug 7 23:08:53.467576092 (13h29m15.211632953s) [upload=13885]: Waiting for error group managing segments... Aug 7 23:08:53.467621884 (13h29m15.211678733s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:53.467627561 (13h29m15.211684422s) [upload=13885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:53.467631218 (13h29m15.211688072s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:53.630342089 (13h29m15.374398952s) [upload=13885]: Done uploading segments... Aug 7 23:08:53.630366467 (13h29m15.37442333s) [upload=13885]: Upload finished. err: <nil> Aug 7 23:10:03.085243036 (13h30m24.829299916s) [upload=13904]: Starting upload Aug 7 23:10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Duration: 874.199383ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [XXX.......................................................XX] duration: 1m0.000792085s FAIL (2025-08-07 23:12:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5.087944464 (13h22m16.832001314s) [upload=13761]: Upload finished. err: <nil> Aug 7 23:02:35.452281267 (13h22m57.196338128s) [upload=13772]: Starting upload Aug 7 23:02:35.452363022 (13h22m57.196419874s) [upload=13772]: Done starting upload Aug 7 23:02:35.452463263 (13h22m57.196520113s) [upload=13772]: Uploading segments... Aug 7 23:02:35.452523098 (13h22m57.196579966s) [upload=13772]: Got next segment. Inline: true Aug 7 23:02:35.452528530 (13h22m57.19658539s) [upload=13772]: Waiting for error group managing segments... Aug 7 23:02:35.452574263 (13h22m57.196631123s) [upload=13772]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:02:35.452579819 (13h22m57.196636679s) [upload=13772]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:02:35.452583343 (13h22m57.196640204s) [upload=13772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:02:35.594791272 (13h22m57.338848123s) [upload=13772]: Done uploading segments... Aug 7 23:02:35.594825533 (13h22m57.338882393s) [upload=13772]: Upload finished. err: <nil> Aug 7 23:03:49.636350992 (13h24m11.380407843s) [upload=13795]: Starting upload Aug 7 23:03:49.636417823 (13h24m11.380474677s) [upload=13795]: Done starting upload Aug 7 23:03:49.636490213 (13h24m11.380547076s) [upload=13795]: Uploading segments... Aug 7 23:03:49.636567311 (13h24m11.380624185s) [upload=13795]: Got next segment. Inline: true Aug 7 23:03:49.636574436 (13h24m11.380631287s) [upload=13795]: Waiting for error group managing segments... Aug 7 23:03:49.636645517 (13h24m11.380702374s) [upload=13795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:03:49.636655113 (13h24m11.380711973s) [upload=13795]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:03:49.636658487 (13h24m11.380715343s) [upload=13795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:03:49.781691324 (13h24m11.525748175s) [upload=13795]: Done uploading segments... Aug 7 23:03:49.781713904 (13h24m11.52577076s) [upload=13795]: Upload finished. err: <nil> Aug 7 23:04:57.312274559 (13h25m19.05633143s) [upload=13814]: Starting upload Aug 7 23:04:57.312366372 (13h25m19.056423226s) [upload=13814]: Done starting upload Aug 7 23:04:57.312497707 (13h25m19.056554555s) [upload=13814]: Uploading segments... Aug 7 23:04:57.312567762 (13h25m19.05662463s) [upload=13814]: Got next segment. Inline: true Aug 7 23:04:57.312574196 (13h25m19.056631067s) [upload=13814]: Waiting for error group managing segments... Aug 7 23:04:57.312653597 (13h25m19.056710478s) [upload=13814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:04:57.312660409 (13h25m19.056717278s) [upload=13814]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:04:57.312664013 (13h25m19.056720884s) [upload=13814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:04:57.459986021 (13h25m19.20404287s) [upload=13814]: Done uploading segments... Aug 7 23:04:57.460005087 (13h25m19.204061937s) [upload=13814]: Upload finished. err: <nil> Aug 7 23:05:47.352420719 (13h26m9.096477638s) [upload=13831]: Starting upload Aug 7 23:05:47.352495951 (13h26m9.096552804s) [upload=13831]: Done starting upload Aug 7 23:05:47.352578458 (13h26m9.096635308s) [upload=13831]: Uploading segments... Aug 7 23:05:47.352621273 (13h26m9.096678143s) [upload=13831]: Got next segment. Inline: true Aug 7 23:05:47.352626460 (13h26m9.09668331s) [upload=13831]: Waiting for error group managing segments... Aug 7 23:05:47.352677569 (13h26m9.096734418s) [upload=13831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:05:47.352682758 (13h26m9.096739608s) [upload=13831]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:47.352686503 (13h26m9.096743353s) [upload=13831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:47.525022266 (13h26m9.26907913s) [upload=13831]: Done uploading segments... Aug 7 23:05:47.525047633 (13h26m9.269104482s) [upload=13831]: Upload finished. err: <nil> Aug 7 23:06:58.801209002 (13h27m20.545265866s) [upload=13850]: Starting upload Aug 7 23:06:58.801309642 (13h27m20.545366507s) [upload=13850]: Done starting upload Aug 7 23:06:58.801447578 (13h27m20.545504443s) [upload=13850]: Uploading segments... Aug 7 23:06:58.801538129 (13h27m20.545594996s) [upload=13850]: Got next segment. Inline: true Aug 7 23:06:58.801545498 (13h27m20.54560237s) [upload=13850]: Waiting for error group managing segments... Aug 7 23:06:58.801617944 (13h27m20.545674808s) [upload=13850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:58.801623960 (13h27m20.545680824s) [upload=13850]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:58.801630296 (13h27m20.545687164s) [upload=13850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:58.963809105 (13h27m20.707865989s) [upload=13850]: Done uploading segments... Aug 7 23:06:58.963847876 (13h27m20.707904747s) [upload=13850]: Upload finished. err: <nil> Aug 7 23:07:55.130879231 (13h28m16.874936092s) [upload=13867]: Starting upload Aug 7 23:07:55.130952807 (13h28m16.875009671s) [upload=13867]: Done starting upload Aug 7 23:07:55.131026070 (13h28m16.87508292s) [upload=13867]: Uploading segments... Aug 7 23:07:55.131220158 (13h28m16.875277027s) [upload=13867]: Got next segment. Inline: true Aug 7 23:07:55.131228194 (13h28m16.875285061s) [upload=13867]: Waiting for error group managing segments... Aug 7 23:07:55.131289715 (13h28m16.875346575s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:55.131295462 (13h28m16.875352322s) [upload=13867]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:55.131300279 (13h28m16.875357151s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:55.281757492 (13h28m17.025814344s) [upload=13867]: Done uploading segments... Aug 7 23:07:55.281776844 (13h28m17.025833694s) [upload=13867]: Upload finished. err: <nil> Aug 7 23:08:53.467366061 (13h29m15.211422912s) [upload=13885]: Starting upload Aug 7 23:08:53.467436784 (13h29m15.211493636s) [upload=13885]: Done starting upload Aug 7 23:08:53.467521314 (13h29m15.211578165s) [upload=13885]: Uploading segments... Aug 7 23:08:53.467570153 (13h29m15.211627016s) [upload=13885]: Got next segment. Inline: true Aug 7 23:08:53.467576092 (13h29m15.211632953s) [upload=13885]: Waiting for error group managing segments... Aug 7 23:08:53.467621884 (13h29m15.211678733s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:53.467627561 (13h29m15.211684422s) [upload=13885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:53.467631218 (13h29m15.211688072s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:53.630342089 (13h29m15.374398952s) [upload=13885]: Done uploading segments... Aug 7 23:08:53.630366467 (13h29m15.37442333s) [upload=13885]: Upload finished. err: <nil> Aug 7 23:10:03.085243036 (13h30m24.829299916s) [upload=13904]: Starting upload Aug 7 23:10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:11:40.667902567 (13h32m2.411959413s) [upload=13933]: Starting upload Aug 7 23:11:40.667961940 (13h32m2.412018788s) [upload=13933]: Done starting upload Aug 7 23:11:40.668020524 (13h32m2.41207737s) [upload=13933]: Uploading segments... Aug 7 23:11:40.668075247 (13h32m2.412132113s) [upload=13933]: Got next segment. Inline: true Aug 7 23:11:40.668080664 (13h32m2.412137525s) [upload=13933]: Waiting for error group managing segments... Aug 7 23:11:40.668121190 (13h32m2.412178036s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:40.668125756 (13h32m2.412182616s) [upload=13933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:40.668128431 (13h32m2.412185291s) [upload=13933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:40.394243254 (13h33m2.138300118s) [upload=13933]: Done uploading segments... Aug 7 23:12:40.394270270 (13h33m2.138327132s) [upload=13933]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000810264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...............................X....X............X.......] duration: 5.655700827s FAIL (2025-08-07 23:12:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 5.655735626s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 3.415255474s FAIL (2025-08-07 23:12:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s... Aug 7 22:21:14.636082751 (12h41m36.380139602s) [upload=13066]: Got next segment. Inline: false Aug 7 22:21:14.636136774 (12h41m36.380193634s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:14.636142452 (12h41m36.380199312s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:16.211357551 (12h41m37.955414411s) [seg_pos={0 0}, upload=13066]: Begin upload segment... Aug 7 22:21:16.211426917 (12h41m37.955483784s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214054689 (12h41m37.95811156s) [seg_pos={0 0}, upload=13066]: Done begin upload segment. Aug 7 22:21:16.214143921 (12h41m37.958200791s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214660710 (12h41m37.95871758s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215094607 (12h41m37.959151469s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215554381 (12h41m37.959611251s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215879389 (12h41m37.959936254s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216161781 (12h41m37.960218643s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216489433 (12h41m37.960546284s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216758002 (12h41m37.960814865s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217037043 (12h41m37.961093908s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217313215 (12h41m37.961370077s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217587290 (12h41m37.961644153s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217852877 (12h41m37.961909742s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218138868 (12h41m37.962195731s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218358245 (12h41m37.962415106s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218693596 (12h41m37.962750459s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218971752 (12h41m37.963028615s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.219339785 (12h41m37.963396657s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.820211995 (12h41m38.564268846s) [upload=13066]: Next returned nil segment Aug 7 22:21:16.820729442 (12h41m38.564786312s) [upload=13066]: Waiting for error group managing segments... Aug 7 22:21:16.974534776 (12h41m38.71859164s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:16.995935209 (12h41m38.739992059s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.005410191 (12h41m38.749467042s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.012303228 (12h41m38.756360079s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.047899041 (12h41m38.791955903s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.065632282 (12h41m38.809689142s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.074330593 (12h41m38.818387457s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.205659931 (12h41m38.949716794s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.304344891 (12h41m39.048401752s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.476320837 (12h41m39.220377687s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.498349653 (12h41m39.242406517s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.566667384 (12h41m39.310724234s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.572654381 (12h41m39.316711251s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002510167 (12h41m39.746567013s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002534446 (12h41m39.746591307s) [seg_pos={0 0}, upload=13066]: Segment reached optimal threshold of 14 pieces. Aug 7 22:21:18.002684920 (12h41m39.746741766s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12goRcYW39Qqga4HW1i6cg6VGKJFUxnyTJBe6HRnZiLAU3LonWn): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002753282 (12h41m39.746810128s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002945708 (12h41m39.747002555s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.003059994 (12h41m39.74711684s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection Aug 7 22:21:18.003080731 (12h41m39.747137592s) [seg_pos={0 0}, upload=13066]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:21:18.003298604 (12h41m39.747355464s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:21:18.003304952 (12h41m39.747361812s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:15.962677772 (12h50m37.706734624s) [upload=13066]: Done uploading segments... Aug 7 22:30:15.962709891 (12h50m37.706766753s) [upload=13066]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.415298524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................................X] duration: 30.000652071s FAIL (2025-08-07 23:12:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Aug 7 23:03:53.742994904 (13h24m15.487051754s) [upload=13797]: Upload finished. err: <nil> Aug 7 23:05:05.450781453 (13h25m27.194838313s) [upload=13819]: Starting upload Aug 7 23:05:05.450866009 (13h25m27.194922864s) [upload=13819]: Done starting upload Aug 7 23:05:05.450958818 (13h25m27.195015667s) [upload=13819]: Uploading segments... Aug 7 23:05:05.451012819 (13h25m27.195069683s) [upload=13819]: Got next segment. Inline: true Aug 7 23:05:05.451190424 (13h25m27.195247287s) [upload=13819]: Waiting for error group managing segments... Aug 7 23:05:05.451260192 (13h25m27.195317052s) [upload=13819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:05:05.451265767 (13h25m27.195322628s) [upload=13819]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:05.451269125 (13h25m27.195325978s) [upload=13819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:05.614255373 (13h25m27.358312237s) [upload=13819]: Done uploading segments... Aug 7 23:05:05.614274716 (13h25m27.358331571s) [upload=13819]: Upload finished. err: <nil> Aug 7 23:05:37.597719431 (13h25m59.341776283s) [upload=13827]: Starting upload Aug 7 23:05:37.597786920 (13h25m59.341843774s) [upload=13827]: Done starting upload Aug 7 23:05:37.597872453 (13h25m59.341929305s) [upload=13827]: Uploading segments... Aug 7 23:05:37.597954792 (13h25m59.342011657s) [upload=13827]: Got next segment. Inline: true Aug 7 23:05:37.597960599 (13h25m59.342017459s) [upload=13827]: Waiting for error group managing segments... Aug 7 23:05:37.598010394 (13h25m59.342067243s) [upload=13827]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:05:37.598015744 (13h25m59.342072605s) [upload=13827]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:37.598019424 (13h25m59.342076283s) [upload=13827]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:37.749090862 (13h25m59.493147712s) [upload=13827]: Done uploading segments... Aug 7 23:05:37.749110397 (13h25m59.493167247s) [upload=13827]: Upload finished. err: <nil> Aug 7 23:06:41.828111526 (13h27m3.572168391s) [upload=13842]: Starting upload Aug 7 23:06:41.828216224 (13h27m3.572273076s) [upload=13842]: Done starting upload Aug 7 23:06:41.828334659 (13h27m3.572391512s) [upload=13842]: Uploading segments... Aug 7 23:06:41.828449287 (13h27m3.572506151s) [upload=13842]: Got next segment. Inline: true Aug 7 23:06:41.828457932 (13h27m3.572514797s) [upload=13842]: Waiting for error group managing segments... Aug 7 23:06:41.828522114 (13h27m3.572578976s) [upload=13842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:41.828529109 (13h27m3.57258597s) [upload=13842]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:41.828532670 (13h27m3.572589532s) [upload=13842]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:41.979430299 (13h27m3.72348716s) [upload=13842]: Done uploading segments... Aug 7 23:06:41.979453508 (13h27m3.723510358s) [upload=13842]: Upload finished. err: <nil> Aug 7 23:07:32.461706064 (13h27m54.205762928s) [upload=13858]: Starting upload Aug 7 23:07:32.461824536 (13h27m54.205881399s) [upload=13858]: Done starting upload Aug 7 23:07:32.461961765 (13h27m54.206018623s) [upload=13858]: Uploading segments... Aug 7 23:07:32.462039185 (13h27m54.206096056s) [upload=13858]: Got next segment. Inline: true Aug 7 23:07:32.462045894 (13h27m54.206102758s) [upload=13858]: Waiting for error group managing segments... Aug 7 23:07:32.462110153 (13h27m54.206167013s) [upload=13858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:32.462116381 (13h27m54.206173242s) [upload=13858]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:32.462119766 (13h27m54.206176626s) [upload=13858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:32.615220579 (13h27m54.359277439s) [upload=13858]: Done uploading segments... Aug 7 23:07:32.615246418 (13h27m54.359303279s) [upload=13858]: Upload finished. err: <nil> Aug 7 23:08:40.493251780 (13h29m2.237308632s) [upload=13880]: Starting upload Aug 7 23:08:40.493307346 (13h29m2.237364211s) [upload=13880]: Done starting upload Aug 7 23:08:40.493377552 (13h29m2.237434398s) [upload=13880]: Uploading segments... Aug 7 23:08:40.493449296 (13h29m2.237506164s) [upload=13880]: Got next segment. Inline: true Aug 7 23:08:40.493457374 (13h29m2.237514224s) [upload=13880]: Waiting for error group managing segments... Aug 7 23:08:40.493501643 (13h29m2.237558494s) [upload=13880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:40.493507544 (13h29m2.237564404s) [upload=13880]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:40.493510790 (13h29m2.23756764s) [upload=13880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:40.636364852 (13h29m2.380421716s) [upload=13880]: Done uploading segments... Aug 7 23:08:40.636386281 (13h29m2.38044313s) [upload=13880]: Upload finished. err: <nil> Aug 7 23:09:01.297603828 (13h29m23.041660691s) [upload=13886]: Starting upload Aug 7 23:09:01.297709608 (13h29m23.04176648s) [upload=13886]: Done starting upload Aug 7 23:09:01.297804098 (13h29m23.041860954s) [upload=13886]: Uploading segments... Aug 7 23:09:01.297882202 (13h29m23.041939076s) [upload=13886]: Got next segment. Inline: true Aug 7 23:09:01.297889366 (13h29m23.041946233s) [upload=13886]: Waiting for error group managing segments... Aug 7 23:09:01.297998616 (13h29m23.04205548s) [upload=13886]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:01.298006016 (13h29m23.042062881s) [upload=13886]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:01.298010262 (13h29m23.042067126s) [upload=13886]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:01.452127466 (13h29m23.196184318s) [upload=13886]: Done uploading segments... Aug 7 23:09:01.452173116 (13h29m23.196229988s) [upload=13886]: Upload finished. err: <nil> Aug 7 23:09:27.855960951 (13h29m49.600017802s) [upload=13891]: Starting upload Aug 7 23:09:27.856047975 (13h29m49.600104835s) [upload=13891]: Done starting upload Aug 7 23:09:27.856128881 (13h29m49.600185732s) [upload=13891]: Uploading segments... Aug 7 23:09:27.856217016 (13h29m49.600273883s) [upload=13891]: Got next segment. Inline: true Aug 7 23:09:27.856223587 (13h29m49.600280447s) [upload=13891]: Waiting for error group managing segments... Aug 7 23:09:27.856283213 (13h29m49.600340073s) [upload=13891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:27.856289411 (13h29m49.600346272s) [upload=13891]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:27.856292931 (13h29m49.600349791s) [upload=13891]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:28.011857390 (13h29m49.75591424s) [upload=13891]: Done uploading segments... Aug 7 23:09:28.011879735 (13h29m49.755936584s) [upload=13891]: Upload finished. err: <nil> Aug 7 23:10:28.012300991 (13h30m49.756357841s) [upload=13913]: Starting upload Aug 7 23:10:28.012378252 (13h30m49.756435112s) [upload=13913]: Done starting upload Aug 7 23:10:28.012443618 (13h30m49.756500468s) [upload=13913]: Uploading segments... Aug 7 23:10:28.012512066 (13h30m49.756568934s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:28.012518615 (13h30m49.756575478s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:28.012569185 (13h30m49.756626036s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:28.012575496 (13h30m49.756632345s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:28.012578877 (13h30m49.756635741s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:28.178423527 (13h30m49.922480378s) [upload=13913]: Done uploading segments... Aug 7 23:10:28.178443696 (13h30m49.922500546s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:16.095264069 (13h31m37.839320941s) [upload=13924]: Starting upload Aug 7 23:11:16.095345689 (13h31m37.83940255s) [upload=13924]: Done starting upload Aug 7 23:11:16.095421525 (13h31m37.839478385s) [upload=13924]: Uploading segments... Aug 7 23:11:16.095460332 (13h31m37.839517201s) [upload=13924]: Got next segment. Inline: true Aug 7 23:11:16.095465792 (13h31m37.839522653s) [upload=13924]: Waiting for error group managing segments... Aug 7 23:11:16.095512445 (13h31m37.839569308s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:16.095517610 (13h31m37.839574471s) [upload=13924]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:16.095521136 (13h31m37.839577996s) [upload=13924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:16.231539768 (13h31m37.975596629s) [upload=13924]: Done uploading segments... Aug 7 23:11:16.231558376 (13h31m37.975615223s) [upload=13924]: Upload finished. err: <nil> Aug 7 23:11:48.329873818 (13h32m10.073930665s) [upload=13936]: Starting upload Aug 7 23:11:48.329969166 (13h32m10.074026026s) [upload=13936]: Done starting upload Aug 7 23:11:48.330046481 (13h32m10.074103328s) [upload=13936]: Uploading segments... Aug 7 23:11:48.330100337 (13h32m10.074157203s) [upload=13936]: Got next segment. Inline: true Aug 7 23:11:48.330105628 (13h32m10.074162488s) [upload=13936]: Waiting for error group managing segments... Aug 7 23:11:48.330144233 (13h32m10.074201079s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.330148601 (13h32m10.074205447s) [upload=13936]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.330151121 (13h32m10.074207967s) [upload=13936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:18.060862417 (13h32m39.804919268s) [upload=13936]: Done uploading segments... Aug 7 23:12:18.060910946 (13h32m39.804967806s) [upload=13936]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000681665s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 946.067835ms FAIL (2025-08-07 23:12:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 946.11119ms core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X.........X.................................................] duration: 999.88931ms FAIL (2025-08-07 23:12:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 999.918483ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXX..X.....................................................] duration: 2.515181459s FAIL (2025-08-07 23:12:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7 22:27:44.139472223 (12h48m5.883529083s) [upload=13160]: Got next segment. Inline: false Aug 7 22:27:44.139508151 (12h48m5.883565011s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:44.139513399 (12h48m5.883570259s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:27:44.245907064 (12h48m5.989963927s) [seg_pos={0 0}, upload=13160]: Begin upload segment... Aug 7 22:27:44.246039735 (12h48m5.990096597s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.246596012 (12h48m5.990652882s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247006251 (12h48m5.991063115s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247197178 (12h48m5.991254041s) [seg_pos={0 0}, upload=13160]: Done begin upload segment. Aug 7 22:27:44.247225152 (12h48m5.991282018s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247751444 (12h48m5.99180831s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247956514 (12h48m5.992013382s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248499391 (12h48m5.992556252s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248606386 (12h48m5.992663249s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249110151 (12h48m5.99316702s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249195913 (12h48m5.993252781s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249587895 (12h48m5.993644759s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249756005 (12h48m5.993812872s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.250801847 (12h48m5.99485871s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251094602 (12h48m5.995151463s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251408354 (12h48m5.995465217s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251693979 (12h48m5.995750839s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.928589281 (12h48m6.672646155s) [upload=13160]: Next returned nil segment Aug 7 22:27:44.928610126 (12h48m6.672666995s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:27:45.055730069 (12h48m6.799786923s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.058230571 (12h48m6.802287421s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.072176193 (12h48m6.816233043s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.076144433 (12h48m6.82020128s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.083969611 (12h48m6.828026457s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.090499988 (12h48m6.834556849s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.129675895 (12h48m6.873732757s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.148504731 (12h48m6.892561581s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.183435281 (12h48m6.927492128s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.188056276 (12h48m6.932113137s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.189109860 (12h48m6.93316672s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.191395584 (12h48m6.93545243s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212167167 (12h48m6.956224014s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212191124 (12h48m6.956247983s) [seg_pos={0 0}, upload=13160]: Segment reached optimal threshold of 13 pieces. Aug 7 22:27:45.212404510 (12h48m6.956461359s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: context canceled; context canceled Aug 7 22:27:45.212444310 (12h48m6.956501156s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection Aug 7 22:27:45.212507508 (12h48m6.956564355s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection Aug 7 22:27:45.212525428 (12h48m6.956582288s) [seg_pos={0 0}, upload=13160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:27:45.212587349 (12h48m6.956644209s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:27:45.212591472 (12h48m6.956648332s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:45.467451329 (12h48m7.21150818s) [upload=13160]: Done uploading segments... Aug 7 22:27:45.467470231 (12h48m7.211527081s) [upload=13160]: Upload finished. err: <nil> Aug 7 22:28:46.300540067 (12h49m8.044596928s) [upload=13178]: Starting upload Aug 7 22:28:46.300629117 (12h49m8.044685979s) [upload=13178]: Done starting upload Aug 7 22:28:46.300797752 (12h49m8.044854603s) [upload=13178]: Uploading segments... Aug 7 22:28:46.301190738 (12h49m8.045247598s) [upload=13178]: Got next segment. Inline: false Aug 7 22:28:46.301268000 (12h49m8.04532486s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:46.301275318 (12h49m8.045332178s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:29:16.009299446 (12h49m37.753356315s) [upload=13178]: Next segment err: context deadline exceeded Aug 7 22:29:16.009371909 (12h49m37.753428759s) [upload=13178]: Done uploading segments... Aug 7 22:29:16.009392178 (12h49m37.753449035s) [upload=13178]: Upload finished. err: context deadline exceeded Duration: 2.515211527s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.538302987s FAIL (2025-08-07 23:11:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.538331439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 866.404331ms FAIL (2025-08-07 23:11:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 866.43351ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X.........X................................................] duration: 833.736287ms FAIL (2025-08-07 23:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 833.7715ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [XXXX.......................................................X] duration: 1m0.001075463s FAIL (2025-08-07 23:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== nts... Aug 7 23:01:30.054402773 (13h21m51.798459638s) [upload=13756]: Upload finished. err: <nil> Aug 7 23:01:54.945532164 (13h22m16.689589011s) [upload=13761]: Starting upload Aug 7 23:01:54.945592779 (13h22m16.689649649s) [upload=13761]: Done starting upload Aug 7 23:01:54.945707500 (13h22m16.689764352s) [upload=13761]: Uploading segments... Aug 7 23:01:54.945763653 (13h22m16.689820512s) [upload=13761]: Got next segment. Inline: true Aug 7 23:01:54.945768982 (13h22m16.689825843s) [upload=13761]: Waiting for error group managing segments... Aug 7 23:01:54.945808899 (13h22m16.689865749s) [upload=13761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:01:54.945814660 (13h22m16.689871524s) [upload=13761]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:01:54.945819600 (13h22m16.689876465s) [upload=13761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:01:55.087924990 (13h22m16.831981841s) [upload=13761]: Done uploading segments... Aug 7 23:01:55.087944464 (13h22m16.832001314s) [upload=13761]: Upload finished. err: <nil> Aug 7 23:02:35.452281267 (13h22m57.196338128s) [upload=13772]: Starting upload Aug 7 23:02:35.452363022 (13h22m57.196419874s) [upload=13772]: Done starting upload Aug 7 23:02:35.452463263 (13h22m57.196520113s) [upload=13772]: Uploading segments... Aug 7 23:02:35.452523098 (13h22m57.196579966s) [upload=13772]: Got next segment. Inline: true Aug 7 23:02:35.452528530 (13h22m57.19658539s) [upload=13772]: Waiting for error group managing segments... Aug 7 23:02:35.452574263 (13h22m57.196631123s) [upload=13772]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:02:35.452579819 (13h22m57.196636679s) [upload=13772]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:02:35.452583343 (13h22m57.196640204s) [upload=13772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:02:35.594791272 (13h22m57.338848123s) [upload=13772]: Done uploading segments... Aug 7 23:02:35.594825533 (13h22m57.338882393s) [upload=13772]: Upload finished. err: <nil> Aug 7 23:03:49.636350992 (13h24m11.380407843s) [upload=13795]: Starting upload Aug 7 23:03:49.636417823 (13h24m11.380474677s) [upload=13795]: Done starting upload Aug 7 23:03:49.636490213 (13h24m11.380547076s) [upload=13795]: Uploading segments... Aug 7 23:03:49.636567311 (13h24m11.380624185s) [upload=13795]: Got next segment. Inline: true Aug 7 23:03:49.636574436 (13h24m11.380631287s) [upload=13795]: Waiting for error group managing segments... Aug 7 23:03:49.636645517 (13h24m11.380702374s) [upload=13795]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:03:49.636655113 (13h24m11.380711973s) [upload=13795]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:03:49.636658487 (13h24m11.380715343s) [upload=13795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:03:49.781691324 (13h24m11.525748175s) [upload=13795]: Done uploading segments... Aug 7 23:03:49.781713904 (13h24m11.52577076s) [upload=13795]: Upload finished. err: <nil> Aug 7 23:04:57.312274559 (13h25m19.05633143s) [upload=13814]: Starting upload Aug 7 23:04:57.312366372 (13h25m19.056423226s) [upload=13814]: Done starting upload Aug 7 23:04:57.312497707 (13h25m19.056554555s) [upload=13814]: Uploading segments... Aug 7 23:04:57.312567762 (13h25m19.05662463s) [upload=13814]: Got next segment. Inline: true Aug 7 23:04:57.312574196 (13h25m19.056631067s) [upload=13814]: Waiting for error group managing segments... Aug 7 23:04:57.312653597 (13h25m19.056710478s) [upload=13814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:04:57.312660409 (13h25m19.056717278s) [upload=13814]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:04:57.312664013 (13h25m19.056720884s) [upload=13814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:04:57.459986021 (13h25m19.20404287s) [upload=13814]: Done uploading segments... Aug 7 23:04:57.460005087 (13h25m19.204061937s) [upload=13814]: Upload finished. err: <nil> Aug 7 23:05:47.352420719 (13h26m9.096477638s) [upload=13831]: Starting upload Aug 7 23:05:47.352495951 (13h26m9.096552804s) [upload=13831]: Done starting upload Aug 7 23:05:47.352578458 (13h26m9.096635308s) [upload=13831]: Uploading segments... Aug 7 23:05:47.352621273 (13h26m9.096678143s) [upload=13831]: Got next segment. Inline: true Aug 7 23:05:47.352626460 (13h26m9.09668331s) [upload=13831]: Waiting for error group managing segments... Aug 7 23:05:47.352677569 (13h26m9.096734418s) [upload=13831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:05:47.352682758 (13h26m9.096739608s) [upload=13831]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:47.352686503 (13h26m9.096743353s) [upload=13831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:05:47.525022266 (13h26m9.26907913s) [upload=13831]: Done uploading segments... Aug 7 23:05:47.525047633 (13h26m9.269104482s) [upload=13831]: Upload finished. err: <nil> Aug 7 23:06:58.801209002 (13h27m20.545265866s) [upload=13850]: Starting upload Aug 7 23:06:58.801309642 (13h27m20.545366507s) [upload=13850]: Done starting upload Aug 7 23:06:58.801447578 (13h27m20.545504443s) [upload=13850]: Uploading segments... Aug 7 23:06:58.801538129 (13h27m20.545594996s) [upload=13850]: Got next segment. Inline: true Aug 7 23:06:58.801545498 (13h27m20.54560237s) [upload=13850]: Waiting for error group managing segments... Aug 7 23:06:58.801617944 (13h27m20.545674808s) [upload=13850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:58.801623960 (13h27m20.545680824s) [upload=13850]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:58.801630296 (13h27m20.545687164s) [upload=13850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:58.963809105 (13h27m20.707865989s) [upload=13850]: Done uploading segments... Aug 7 23:06:58.963847876 (13h27m20.707904747s) [upload=13850]: Upload finished. err: <nil> Aug 7 23:07:55.130879231 (13h28m16.874936092s) [upload=13867]: Starting upload Aug 7 23:07:55.130952807 (13h28m16.875009671s) [upload=13867]: Done starting upload Aug 7 23:07:55.131026070 (13h28m16.87508292s) [upload=13867]: Uploading segments... Aug 7 23:07:55.131220158 (13h28m16.875277027s) [upload=13867]: Got next segment. Inline: true Aug 7 23:07:55.131228194 (13h28m16.875285061s) [upload=13867]: Waiting for error group managing segments... Aug 7 23:07:55.131289715 (13h28m16.875346575s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:55.131295462 (13h28m16.875352322s) [upload=13867]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:55.131300279 (13h28m16.875357151s) [upload=13867]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:55.281757492 (13h28m17.025814344s) [upload=13867]: Done uploading segments... Aug 7 23:07:55.281776844 (13h28m17.025833694s) [upload=13867]: Upload finished. err: <nil> Aug 7 23:08:53.467366061 (13h29m15.211422912s) [upload=13885]: Starting upload Aug 7 23:08:53.467436784 (13h29m15.211493636s) [upload=13885]: Done starting upload Aug 7 23:08:53.467521314 (13h29m15.211578165s) [upload=13885]: Uploading segments... Aug 7 23:08:53.467570153 (13h29m15.211627016s) [upload=13885]: Got next segment. Inline: true Aug 7 23:08:53.467576092 (13h29m15.211632953s) [upload=13885]: Waiting for error group managing segments... Aug 7 23:08:53.467621884 (13h29m15.211678733s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:53.467627561 (13h29m15.211684422s) [upload=13885]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:53.467631218 (13h29m15.211688072s) [upload=13885]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:53.630342089 (13h29m15.374398952s) [upload=13885]: Done uploading segments... Aug 7 23:08:53.630366467 (13h29m15.37442333s) [upload=13885]: Upload finished. err: <nil> Aug 7 23:10:03.085243036 (13h30m24.829299916s) [upload=13904]: Starting upload Aug 7 23:10:03.085332650 (13h30m24.829389513s) [upload=13904]: Done starting upload Aug 7 23:10:03.085414170 (13h30m24.829471025s) [upload=13904]: Uploading segments... Aug 7 23:10:03.085475435 (13h30m24.829532308s) [upload=13904]: Got next segment. Inline: true Aug 7 23:10:03.085480843 (13h30m24.829537706s) [upload=13904]: Waiting for error group managing segments... Aug 7 23:10:03.085544347 (13h30m24.829601211s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.085552551 (13h30m24.829609424s) [upload=13904]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.085555974 (13h30m24.829612844s) [upload=13904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.247308964 (13h30m24.991365825s) [upload=13904]: Done uploading segments... Aug 7 23:10:03.247396260 (13h30m24.991453126s) [upload=13904]: Upload finished. err: <nil> Aug 7 23:10:40.522399484 (13h31m2.266456348s) [upload=13915]: Starting upload Aug 7 23:10:40.522487646 (13h31m2.266544496s) [upload=13915]: Done starting upload Aug 7 23:10:40.522571041 (13h31m2.266627902s) [upload=13915]: Uploading segments... Aug 7 23:10:40.522619425 (13h31m2.266676294s) [upload=13915]: Got next segment. Inline: true Aug 7 23:10:40.522626307 (13h31m2.266683183s) [upload=13915]: Waiting for error group managing segments... Aug 7 23:10:40.522684106 (13h31m2.266740971s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:40.522691484 (13h31m2.26674835s) [upload=13915]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:40.522696090 (13h31m2.266752961s) [upload=13915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:40.254283342 (13h32m1.998340192s) [upload=13915]: Done uploading segments... Aug 7 23:11:40.254304548 (13h32m1.998361398s) [upload=13915]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001098969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 3.221286439s FAIL (2025-08-07 23:11:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s... Aug 7 22:21:14.636082751 (12h41m36.380139602s) [upload=13066]: Got next segment. Inline: false Aug 7 22:21:14.636136774 (12h41m36.380193634s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:14.636142452 (12h41m36.380199312s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:16.211357551 (12h41m37.955414411s) [seg_pos={0 0}, upload=13066]: Begin upload segment... Aug 7 22:21:16.211426917 (12h41m37.955483784s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214054689 (12h41m37.95811156s) [seg_pos={0 0}, upload=13066]: Done begin upload segment. Aug 7 22:21:16.214143921 (12h41m37.958200791s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214660710 (12h41m37.95871758s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215094607 (12h41m37.959151469s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215554381 (12h41m37.959611251s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215879389 (12h41m37.959936254s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216161781 (12h41m37.960218643s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216489433 (12h41m37.960546284s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216758002 (12h41m37.960814865s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217037043 (12h41m37.961093908s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217313215 (12h41m37.961370077s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217587290 (12h41m37.961644153s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217852877 (12h41m37.961909742s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218138868 (12h41m37.962195731s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218358245 (12h41m37.962415106s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218693596 (12h41m37.962750459s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218971752 (12h41m37.963028615s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.219339785 (12h41m37.963396657s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.820211995 (12h41m38.564268846s) [upload=13066]: Next returned nil segment Aug 7 22:21:16.820729442 (12h41m38.564786312s) [upload=13066]: Waiting for error group managing segments... Aug 7 22:21:16.974534776 (12h41m38.71859164s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:16.995935209 (12h41m38.739992059s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.005410191 (12h41m38.749467042s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.012303228 (12h41m38.756360079s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.047899041 (12h41m38.791955903s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.065632282 (12h41m38.809689142s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.074330593 (12h41m38.818387457s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.205659931 (12h41m38.949716794s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.304344891 (12h41m39.048401752s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.476320837 (12h41m39.220377687s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.498349653 (12h41m39.242406517s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.566667384 (12h41m39.310724234s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.572654381 (12h41m39.316711251s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002510167 (12h41m39.746567013s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002534446 (12h41m39.746591307s) [seg_pos={0 0}, upload=13066]: Segment reached optimal threshold of 14 pieces. Aug 7 22:21:18.002684920 (12h41m39.746741766s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12goRcYW39Qqga4HW1i6cg6VGKJFUxnyTJBe6HRnZiLAU3LonWn): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002753282 (12h41m39.746810128s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002945708 (12h41m39.747002555s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.003059994 (12h41m39.74711684s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection Aug 7 22:21:18.003080731 (12h41m39.747137592s) [seg_pos={0 0}, upload=13066]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:21:18.003298604 (12h41m39.747355464s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:21:18.003304952 (12h41m39.747361812s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:15.962677772 (12h50m37.706734624s) [upload=13066]: Done uploading segments... Aug 7 22:30:15.962709891 (12h50m37.706766753s) [upload=13066]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.221314797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [X.XX........................................................] duration: 903.610657ms FAIL (2025-08-07 23:11:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 903.662095ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...............................X....X............X......] duration: 3.11834418s FAIL (2025-08-07 23:11:31 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.118373986s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.........X...............................................] duration: 945.103829ms FAIL (2025-08-07 23:11:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 945.145947ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.713082188s FAIL (2025-08-07 23:11:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.713113276s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 860.421228ms FAIL (2025-08-07 23:11:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=13077]: Starting upload Aug 7 22:22:03.530157956 (12h42m25.27421483s) [upload=13077]: Done starting upload Aug 7 22:22:03.530254340 (12h42m25.274311199s) [upload=13077]: Uploading segments... Aug 7 22:22:03.530319521 (12h42m25.274376398s) [upload=13077]: Got next segment. Inline: true Aug 7 22:22:03.530326839 (12h42m25.27438371s) [upload=13077]: Waiting for error group managing segments... Aug 7 22:22:03.530387033 (12h42m25.274443903s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:03.530393764 (12h42m25.274450635s) [upload=13077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:03.530398068 (12h42m25.274454934s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:03.668101987 (12h42m25.412158856s) [upload=13077]: Done uploading segments... Aug 7 22:22:03.668146181 (12h42m25.412203031s) [upload=13077]: Upload finished. err: <nil> Aug 7 22:22:50.781175997 (12h43m12.525232844s) [upload=13085]: Starting upload Aug 7 22:22:50.781238492 (12h43m12.525295338s) [upload=13085]: Done starting upload Aug 7 22:22:50.781295832 (12h43m12.525352678s) [upload=13085]: Uploading segments... Aug 7 22:22:50.781339974 (12h43m12.525396844s) [upload=13085]: Got next segment. Inline: true Aug 7 22:22:50.781344619 (12h43m12.525401479s) [upload=13085]: Waiting for error group managing segments... Aug 7 22:22:50.781382529 (12h43m12.525439375s) [upload=13085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:50.781387267 (12h43m12.525444113s) [upload=13085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:50.781389841 (12h43m12.525446704s) [upload=13085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:20.513298723 (12h43m42.257355587s) [upload=13085]: Done uploading segments... Aug 7 22:23:20.513325654 (12h43m42.25738252s) [upload=13085]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:51.254516692 (12h44m12.998573555s) [upload=13099]: Starting upload Aug 7 22:23:51.254606307 (12h44m12.998663169s) [upload=13099]: Done starting upload Aug 7 22:23:51.254669521 (12h44m12.998726371s) [upload=13099]: Uploading segments... Aug 7 22:23:51.254709863 (12h44m12.998766733s) [upload=13099]: Got next segment. Inline: true Aug 7 22:23:51.254715030 (12h44m12.99877189s) [upload=13099]: Waiting for error group managing segments... Aug 7 22:23:51.254768832 (12h44m12.998825693s) [upload=13099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:51.254774395 (12h44m12.998831255s) [upload=13099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:51.254778260 (12h44m12.998835121s) [upload=13099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:20.980586635 (12h44m42.724643486s) [upload=13099]: Done uploading segments... Aug 7 22:24:20.980611487 (12h44m42.724668337s) [upload=13099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:22.809871679 (12h44m44.55392853s) [upload=13108]: Starting upload Aug 7 22:24:22.809959685 (12h44m44.554016545s) [upload=13108]: Done starting upload Aug 7 22:24:22.810026016 (12h44m44.55408288s) [upload=13108]: Uploading segments... Aug 7 22:24:22.810084984 (12h44m44.554141856s) [upload=13108]: Got next segment. Inline: true Aug 7 22:24:22.810090937 (12h44m44.554147807s) [upload=13108]: Waiting for error group managing segments... Aug 7 22:24:22.810180251 (12h44m44.554237121s) [upload=13108]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:22.810187976 (12h44m44.554244835s) [upload=13108]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:22.810192661 (12h44m44.55424953s) [upload=13108]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:52.535275009 (12h45m14.279331864s) [upload=13108]: Done uploading segments... Aug 7 22:24:52.535303649 (12h45m14.279360515s) [upload=13108]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:52.960601883 (12h45m14.70465875s) [upload=13112]: Starting upload Aug 7 22:24:52.960708121 (12h45m14.704764984s) [upload=13112]: Done starting upload Aug 7 22:24:52.960845573 (12h45m14.704902419s) [upload=13112]: Uploading segments... Aug 7 22:24:52.960931513 (12h45m14.704988384s) [upload=13112]: Got next segment. Inline: true Aug 7 22:24:52.960939376 (12h45m14.704996243s) [upload=13112]: Waiting for error group managing segments... Aug 7 22:24:52.961019433 (12h45m14.705076313s) [upload=13112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:52.961026755 (12h45m14.705083622s) [upload=13112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:52.961030905 (12h45m14.705087769s) [upload=13112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:22.687567107 (12h45m44.431623957s) [upload=13112]: Done uploading segments... Aug 7 22:25:22.687590061 (12h45m44.431646912s) [upload=13112]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:23.131013864 (12h45m44.875070712s) [upload=13122]: Starting upload Aug 7 22:25:23.131088619 (12h45m44.875145482s) [upload=13122]: Done starting upload Aug 7 22:25:23.131275088 (12h45m44.875331941s) [upload=13122]: Uploading segments... Aug 7 22:25:23.131376992 (12h45m44.875433843s) [upload=13122]: Got next segment. Inline: true Aug 7 22:25:23.131382965 (12h45m44.875439816s) [upload=13122]: Waiting for error group managing segments... Aug 7 22:25:23.131434882 (12h45m44.875491733s) [upload=13122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:23.131440120 (12h45m44.875496982s) [upload=13122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:23.131443668 (12h45m44.875500519s) [upload=13122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:23.272670449 (12h45m45.016727299s) [upload=13122]: Done uploading segments... Aug 7 22:25:23.272702241 (12h45m45.016759091s) [upload=13122]: Upload finished. err: <nil> Aug 7 22:26:25.811198326 (12h46m47.555255193s) [upload=13140]: Starting upload Aug 7 22:26:25.811285415 (12h46m47.555342279s) [upload=13140]: Done starting upload Aug 7 22:26:25.811369732 (12h46m47.555426582s) [upload=13140]: Uploading segments... Aug 7 22:26:25.811467114 (12h46m47.55552398s) [upload=13140]: Got next segment. Inline: true Aug 7 22:26:25.811473701 (12h46m47.555530552s) [upload=13140]: Waiting for error group managing segments... Aug 7 22:26:25.811539126 (12h46m47.555595979s) [upload=13140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:25.811545282 (12h46m47.555602146s) [upload=13140]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:25.811548939 (12h46m47.555605802s) [upload=13140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:25.968679249 (12h46m47.712736113s) [upload=13140]: Done uploading segments... Aug 7 22:26:25.968709074 (12h46m47.712765928s) [upload=13140]: Upload finished. err: <nil> Aug 7 22:27:27.799853559 (12h47m49.543910409s) [upload=13154]: Starting upload Aug 7 22:27:27.799998197 (12h47m49.544055055s) [upload=13154]: Done starting upload Aug 7 22:27:27.800097878 (12h47m49.544154732s) [upload=13154]: Uploading segments... Aug 7 22:27:27.800202613 (12h47m49.544259463s) [upload=13154]: Got next segment. Inline: true Aug 7 22:27:27.800221436 (12h47m49.544278297s) [upload=13154]: Waiting for error group managing segments... Aug 7 22:27:27.800305006 (12h47m49.544361867s) [upload=13154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:27.800310790 (12h47m49.54436765s) [upload=13154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:27.800314678 (12h47m49.544371539s) [upload=13154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:27.950105257 (12h47m49.694162117s) [upload=13154]: Done uploading segments... Aug 7 22:27:27.950124372 (12h47m49.694181233s) [upload=13154]: Upload finished. err: <nil> Aug 7 22:27:59.550898410 (12h48m21.29495526s) [upload=13165]: Starting upload Aug 7 22:27:59.550970829 (12h48m21.295027691s) [upload=13165]: Done starting upload Aug 7 22:27:59.551049380 (12h48m21.295106232s) [upload=13165]: Uploading segments... Aug 7 22:27:59.551283044 (12h48m21.295339906s) [upload=13165]: Got next segment. Inline: true Aug 7 22:27:59.551293209 (12h48m21.295350069s) [upload=13165]: Waiting for error group managing segments... Aug 7 22:27:59.551359587 (12h48m21.295416438s) [upload=13165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:59.551365951 (12h48m21.295422812s) [upload=13165]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:59.551369453 (12h48m21.295426314s) [upload=13165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:59.691525895 (12h48m21.435582778s) [upload=13165]: Done uploading segments... Aug 7 22:27:59.691551086 (12h48m21.435607933s) [upload=13165]: Upload finished. err: <nil> Aug 7 22:29:06.342665932 (12h49m28.086722791s) [upload=13183]: Starting upload Aug 7 22:29:06.342756156 (12h49m28.086813016s) [upload=13183]: Done starting upload Aug 7 22:29:06.342875471 (12h49m28.086932321s) [upload=13183]: Uploading segments... Aug 7 22:29:06.343020027 (12h49m28.087076894s) [upload=13183]: Got next segment. Inline: true Aug 7 22:29:06.343029056 (12h49m28.087085926s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:29:06.343276816 (12h49m28.087333721s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:06.343287054 (12h49m28.087343919s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:06.343290895 (12h49m28.08734776s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:36.070785151 (12h49m57.814842003s) [upload=13183]: Done uploading segments... Aug 7 22:29:36.070807148 (12h49m57.814863999s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Duration: 860.51011ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.262939661s FAIL (2025-08-07 23:10:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.262971619s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXX..X....................................................] duration: 2.498240632s FAIL (2025-08-07 23:10:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7 22:27:44.139472223 (12h48m5.883529083s) [upload=13160]: Got next segment. Inline: false Aug 7 22:27:44.139508151 (12h48m5.883565011s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:44.139513399 (12h48m5.883570259s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:27:44.245907064 (12h48m5.989963927s) [seg_pos={0 0}, upload=13160]: Begin upload segment... Aug 7 22:27:44.246039735 (12h48m5.990096597s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.246596012 (12h48m5.990652882s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247006251 (12h48m5.991063115s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247197178 (12h48m5.991254041s) [seg_pos={0 0}, upload=13160]: Done begin upload segment. Aug 7 22:27:44.247225152 (12h48m5.991282018s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247751444 (12h48m5.99180831s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247956514 (12h48m5.992013382s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248499391 (12h48m5.992556252s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248606386 (12h48m5.992663249s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249110151 (12h48m5.99316702s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249195913 (12h48m5.993252781s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249587895 (12h48m5.993644759s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249756005 (12h48m5.993812872s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.250801847 (12h48m5.99485871s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251094602 (12h48m5.995151463s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251408354 (12h48m5.995465217s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251693979 (12h48m5.995750839s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.928589281 (12h48m6.672646155s) [upload=13160]: Next returned nil segment Aug 7 22:27:44.928610126 (12h48m6.672666995s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:27:45.055730069 (12h48m6.799786923s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.058230571 (12h48m6.802287421s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.072176193 (12h48m6.816233043s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.076144433 (12h48m6.82020128s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.083969611 (12h48m6.828026457s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.090499988 (12h48m6.834556849s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.129675895 (12h48m6.873732757s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.148504731 (12h48m6.892561581s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.183435281 (12h48m6.927492128s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.188056276 (12h48m6.932113137s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.189109860 (12h48m6.93316672s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.191395584 (12h48m6.93545243s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212167167 (12h48m6.956224014s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212191124 (12h48m6.956247983s) [seg_pos={0 0}, upload=13160]: Segment reached optimal threshold of 13 pieces. Aug 7 22:27:45.212404510 (12h48m6.956461359s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: context canceled; context canceled Aug 7 22:27:45.212444310 (12h48m6.956501156s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection Aug 7 22:27:45.212507508 (12h48m6.956564355s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection Aug 7 22:27:45.212525428 (12h48m6.956582288s) [seg_pos={0 0}, upload=13160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:27:45.212587349 (12h48m6.956644209s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:27:45.212591472 (12h48m6.956648332s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:45.467451329 (12h48m7.21150818s) [upload=13160]: Done uploading segments... Aug 7 22:27:45.467470231 (12h48m7.211527081s) [upload=13160]: Upload finished. err: <nil> Aug 7 22:28:46.300540067 (12h49m8.044596928s) [upload=13178]: Starting upload Aug 7 22:28:46.300629117 (12h49m8.044685979s) [upload=13178]: Done starting upload Aug 7 22:28:46.300797752 (12h49m8.044854603s) [upload=13178]: Uploading segments... Aug 7 22:28:46.301190738 (12h49m8.045247598s) [upload=13178]: Got next segment. Inline: false Aug 7 22:28:46.301268000 (12h49m8.04532486s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:46.301275318 (12h49m8.045332178s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:29:16.009299446 (12h49m37.753356315s) [upload=13178]: Next segment err: context deadline exceeded Aug 7 22:29:16.009371909 (12h49m37.753428759s) [upload=13178]: Done uploading segments... Aug 7 22:29:16.009392178 (12h49m37.753449035s) [upload=13178]: Upload finished. err: context deadline exceeded Duration: 2.498279789s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.XXXX.X.....................................................] duration: 3.208884529s FAIL (2025-08-07 23:10:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.208912944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration: 3.208787865s FAIL (2025-08-07 23:10:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s... Aug 7 22:21:14.636082751 (12h41m36.380139602s) [upload=13066]: Got next segment. Inline: false Aug 7 22:21:14.636136774 (12h41m36.380193634s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:14.636142452 (12h41m36.380199312s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:16.211357551 (12h41m37.955414411s) [seg_pos={0 0}, upload=13066]: Begin upload segment... Aug 7 22:21:16.211426917 (12h41m37.955483784s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214054689 (12h41m37.95811156s) [seg_pos={0 0}, upload=13066]: Done begin upload segment. Aug 7 22:21:16.214143921 (12h41m37.958200791s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214660710 (12h41m37.95871758s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215094607 (12h41m37.959151469s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215554381 (12h41m37.959611251s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215879389 (12h41m37.959936254s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216161781 (12h41m37.960218643s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216489433 (12h41m37.960546284s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216758002 (12h41m37.960814865s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217037043 (12h41m37.961093908s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217313215 (12h41m37.961370077s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217587290 (12h41m37.961644153s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217852877 (12h41m37.961909742s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218138868 (12h41m37.962195731s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218358245 (12h41m37.962415106s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218693596 (12h41m37.962750459s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218971752 (12h41m37.963028615s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.219339785 (12h41m37.963396657s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.820211995 (12h41m38.564268846s) [upload=13066]: Next returned nil segment Aug 7 22:21:16.820729442 (12h41m38.564786312s) [upload=13066]: Waiting for error group managing segments... Aug 7 22:21:16.974534776 (12h41m38.71859164s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:16.995935209 (12h41m38.739992059s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.005410191 (12h41m38.749467042s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.012303228 (12h41m38.756360079s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.047899041 (12h41m38.791955903s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.065632282 (12h41m38.809689142s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.074330593 (12h41m38.818387457s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.205659931 (12h41m38.949716794s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.304344891 (12h41m39.048401752s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.476320837 (12h41m39.220377687s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.498349653 (12h41m39.242406517s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.566667384 (12h41m39.310724234s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.572654381 (12h41m39.316711251s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002510167 (12h41m39.746567013s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002534446 (12h41m39.746591307s) [seg_pos={0 0}, upload=13066]: Segment reached optimal threshold of 14 pieces. Aug 7 22:21:18.002684920 (12h41m39.746741766s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12goRcYW39Qqga4HW1i6cg6VGKJFUxnyTJBe6HRnZiLAU3LonWn): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002753282 (12h41m39.746810128s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002945708 (12h41m39.747002555s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.003059994 (12h41m39.74711684s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection Aug 7 22:21:18.003080731 (12h41m39.747137592s) [seg_pos={0 0}, upload=13066]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:21:18.003298604 (12h41m39.747355464s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:21:18.003304952 (12h41m39.747361812s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:15.962677772 (12h50m37.706734624s) [upload=13066]: Done uploading segments... Aug 7 22:30:15.962709891 (12h50m37.706766753s) [upload=13066]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.208837669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X.XX.......................................................] duration: 904.306914ms FAIL (2025-08-07 23:10:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 904.340213ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.83162392s FAIL (2025-08-07 23:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.831647967s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...............................X....X............X.....] duration: 3.143127247s FAIL (2025-08-07 23:10:48 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 3.143156737s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.........X..............................................] duration: 850.732989ms FAIL (2025-08-07 23:10:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 850.820009ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 119.736389ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:10:31 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 119.761022ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [X....X......................................................] duration: 1.054246402s FAIL (2025-08-07 23:10:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=13077]: Starting upload Aug 7 22:22:03.530157956 (12h42m25.27421483s) [upload=13077]: Done starting upload Aug 7 22:22:03.530254340 (12h42m25.274311199s) [upload=13077]: Uploading segments... Aug 7 22:22:03.530319521 (12h42m25.274376398s) [upload=13077]: Got next segment. Inline: true Aug 7 22:22:03.530326839 (12h42m25.27438371s) [upload=13077]: Waiting for error group managing segments... Aug 7 22:22:03.530387033 (12h42m25.274443903s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:03.530393764 (12h42m25.274450635s) [upload=13077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:03.530398068 (12h42m25.274454934s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:03.668101987 (12h42m25.412158856s) [upload=13077]: Done uploading segments... Aug 7 22:22:03.668146181 (12h42m25.412203031s) [upload=13077]: Upload finished. err: <nil> Aug 7 22:22:50.781175997 (12h43m12.525232844s) [upload=13085]: Starting upload Aug 7 22:22:50.781238492 (12h43m12.525295338s) [upload=13085]: Done starting upload Aug 7 22:22:50.781295832 (12h43m12.525352678s) [upload=13085]: Uploading segments... Aug 7 22:22:50.781339974 (12h43m12.525396844s) [upload=13085]: Got next segment. Inline: true Aug 7 22:22:50.781344619 (12h43m12.525401479s) [upload=13085]: Waiting for error group managing segments... Aug 7 22:22:50.781382529 (12h43m12.525439375s) [upload=13085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:50.781387267 (12h43m12.525444113s) [upload=13085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:50.781389841 (12h43m12.525446704s) [upload=13085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:20.513298723 (12h43m42.257355587s) [upload=13085]: Done uploading segments... Aug 7 22:23:20.513325654 (12h43m42.25738252s) [upload=13085]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:51.254516692 (12h44m12.998573555s) [upload=13099]: Starting upload Aug 7 22:23:51.254606307 (12h44m12.998663169s) [upload=13099]: Done starting upload Aug 7 22:23:51.254669521 (12h44m12.998726371s) [upload=13099]: Uploading segments... Aug 7 22:23:51.254709863 (12h44m12.998766733s) [upload=13099]: Got next segment. Inline: true Aug 7 22:23:51.254715030 (12h44m12.99877189s) [upload=13099]: Waiting for error group managing segments... Aug 7 22:23:51.254768832 (12h44m12.998825693s) [upload=13099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:51.254774395 (12h44m12.998831255s) [upload=13099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:51.254778260 (12h44m12.998835121s) [upload=13099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:20.980586635 (12h44m42.724643486s) [upload=13099]: Done uploading segments... Aug 7 22:24:20.980611487 (12h44m42.724668337s) [upload=13099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:22.809871679 (12h44m44.55392853s) [upload=13108]: Starting upload Aug 7 22:24:22.809959685 (12h44m44.554016545s) [upload=13108]: Done starting upload Aug 7 22:24:22.810026016 (12h44m44.55408288s) [upload=13108]: Uploading segments... Aug 7 22:24:22.810084984 (12h44m44.554141856s) [upload=13108]: Got next segment. Inline: true Aug 7 22:24:22.810090937 (12h44m44.554147807s) [upload=13108]: Waiting for error group managing segments... Aug 7 22:24:22.810180251 (12h44m44.554237121s) [upload=13108]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:22.810187976 (12h44m44.554244835s) [upload=13108]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:22.810192661 (12h44m44.55424953s) [upload=13108]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:52.535275009 (12h45m14.279331864s) [upload=13108]: Done uploading segments... Aug 7 22:24:52.535303649 (12h45m14.279360515s) [upload=13108]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:52.960601883 (12h45m14.70465875s) [upload=13112]: Starting upload Aug 7 22:24:52.960708121 (12h45m14.704764984s) [upload=13112]: Done starting upload Aug 7 22:24:52.960845573 (12h45m14.704902419s) [upload=13112]: Uploading segments... Aug 7 22:24:52.960931513 (12h45m14.704988384s) [upload=13112]: Got next segment. Inline: true Aug 7 22:24:52.960939376 (12h45m14.704996243s) [upload=13112]: Waiting for error group managing segments... Aug 7 22:24:52.961019433 (12h45m14.705076313s) [upload=13112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:52.961026755 (12h45m14.705083622s) [upload=13112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:52.961030905 (12h45m14.705087769s) [upload=13112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:22.687567107 (12h45m44.431623957s) [upload=13112]: Done uploading segments... Aug 7 22:25:22.687590061 (12h45m44.431646912s) [upload=13112]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:23.131013864 (12h45m44.875070712s) [upload=13122]: Starting upload Aug 7 22:25:23.131088619 (12h45m44.875145482s) [upload=13122]: Done starting upload Aug 7 22:25:23.131275088 (12h45m44.875331941s) [upload=13122]: Uploading segments... Aug 7 22:25:23.131376992 (12h45m44.875433843s) [upload=13122]: Got next segment. Inline: true Aug 7 22:25:23.131382965 (12h45m44.875439816s) [upload=13122]: Waiting for error group managing segments... Aug 7 22:25:23.131434882 (12h45m44.875491733s) [upload=13122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:23.131440120 (12h45m44.875496982s) [upload=13122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:23.131443668 (12h45m44.875500519s) [upload=13122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:23.272670449 (12h45m45.016727299s) [upload=13122]: Done uploading segments... Aug 7 22:25:23.272702241 (12h45m45.016759091s) [upload=13122]: Upload finished. err: <nil> Aug 7 22:26:25.811198326 (12h46m47.555255193s) [upload=13140]: Starting upload Aug 7 22:26:25.811285415 (12h46m47.555342279s) [upload=13140]: Done starting upload Aug 7 22:26:25.811369732 (12h46m47.555426582s) [upload=13140]: Uploading segments... Aug 7 22:26:25.811467114 (12h46m47.55552398s) [upload=13140]: Got next segment. Inline: true Aug 7 22:26:25.811473701 (12h46m47.555530552s) [upload=13140]: Waiting for error group managing segments... Aug 7 22:26:25.811539126 (12h46m47.555595979s) [upload=13140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:25.811545282 (12h46m47.555602146s) [upload=13140]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:25.811548939 (12h46m47.555605802s) [upload=13140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:25.968679249 (12h46m47.712736113s) [upload=13140]: Done uploading segments... Aug 7 22:26:25.968709074 (12h46m47.712765928s) [upload=13140]: Upload finished. err: <nil> Aug 7 22:27:27.799853559 (12h47m49.543910409s) [upload=13154]: Starting upload Aug 7 22:27:27.799998197 (12h47m49.544055055s) [upload=13154]: Done starting upload Aug 7 22:27:27.800097878 (12h47m49.544154732s) [upload=13154]: Uploading segments... Aug 7 22:27:27.800202613 (12h47m49.544259463s) [upload=13154]: Got next segment. Inline: true Aug 7 22:27:27.800221436 (12h47m49.544278297s) [upload=13154]: Waiting for error group managing segments... Aug 7 22:27:27.800305006 (12h47m49.544361867s) [upload=13154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:27.800310790 (12h47m49.54436765s) [upload=13154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:27.800314678 (12h47m49.544371539s) [upload=13154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:27.950105257 (12h47m49.694162117s) [upload=13154]: Done uploading segments... Aug 7 22:27:27.950124372 (12h47m49.694181233s) [upload=13154]: Upload finished. err: <nil> Aug 7 22:27:59.550898410 (12h48m21.29495526s) [upload=13165]: Starting upload Aug 7 22:27:59.550970829 (12h48m21.295027691s) [upload=13165]: Done starting upload Aug 7 22:27:59.551049380 (12h48m21.295106232s) [upload=13165]: Uploading segments... Aug 7 22:27:59.551283044 (12h48m21.295339906s) [upload=13165]: Got next segment. Inline: true Aug 7 22:27:59.551293209 (12h48m21.295350069s) [upload=13165]: Waiting for error group managing segments... Aug 7 22:27:59.551359587 (12h48m21.295416438s) [upload=13165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:59.551365951 (12h48m21.295422812s) [upload=13165]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:59.551369453 (12h48m21.295426314s) [upload=13165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:59.691525895 (12h48m21.435582778s) [upload=13165]: Done uploading segments... Aug 7 22:27:59.691551086 (12h48m21.435607933s) [upload=13165]: Upload finished. err: <nil> Aug 7 22:29:06.342665932 (12h49m28.086722791s) [upload=13183]: Starting upload Aug 7 22:29:06.342756156 (12h49m28.086813016s) [upload=13183]: Done starting upload Aug 7 22:29:06.342875471 (12h49m28.086932321s) [upload=13183]: Uploading segments... Aug 7 22:29:06.343020027 (12h49m28.087076894s) [upload=13183]: Got next segment. Inline: true Aug 7 22:29:06.343029056 (12h49m28.087085926s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:29:06.343276816 (12h49m28.087333721s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:06.343287054 (12h49m28.087343919s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:06.343290895 (12h49m28.08734776s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:36.070785151 (12h49m57.814842003s) [upload=13183]: Done uploading segments... Aug 7 22:29:36.070807148 (12h49m57.814863999s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.054273211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X.........X.............................................] duration: 847.632021ms FAIL (2025-08-07 23:10:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 847.805066ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [X.X.XX......................................................] duration: 914.023808ms FAIL (2025-08-07 23:10:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 914.051964ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXX..X...................................................] duration: 2.75992013s FAIL (2025-08-07 23:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7 22:27:44.139472223 (12h48m5.883529083s) [upload=13160]: Got next segment. Inline: false Aug 7 22:27:44.139508151 (12h48m5.883565011s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:44.139513399 (12h48m5.883570259s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:27:44.245907064 (12h48m5.989963927s) [seg_pos={0 0}, upload=13160]: Begin upload segment... Aug 7 22:27:44.246039735 (12h48m5.990096597s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.246596012 (12h48m5.990652882s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247006251 (12h48m5.991063115s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247197178 (12h48m5.991254041s) [seg_pos={0 0}, upload=13160]: Done begin upload segment. Aug 7 22:27:44.247225152 (12h48m5.991282018s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247751444 (12h48m5.99180831s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247956514 (12h48m5.992013382s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248499391 (12h48m5.992556252s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248606386 (12h48m5.992663249s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249110151 (12h48m5.99316702s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249195913 (12h48m5.993252781s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249587895 (12h48m5.993644759s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249756005 (12h48m5.993812872s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.250801847 (12h48m5.99485871s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251094602 (12h48m5.995151463s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251408354 (12h48m5.995465217s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251693979 (12h48m5.995750839s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.928589281 (12h48m6.672646155s) [upload=13160]: Next returned nil segment Aug 7 22:27:44.928610126 (12h48m6.672666995s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:27:45.055730069 (12h48m6.799786923s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.058230571 (12h48m6.802287421s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.072176193 (12h48m6.816233043s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.076144433 (12h48m6.82020128s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.083969611 (12h48m6.828026457s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.090499988 (12h48m6.834556849s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.129675895 (12h48m6.873732757s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.148504731 (12h48m6.892561581s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.183435281 (12h48m6.927492128s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.188056276 (12h48m6.932113137s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.189109860 (12h48m6.93316672s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.191395584 (12h48m6.93545243s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212167167 (12h48m6.956224014s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212191124 (12h48m6.956247983s) [seg_pos={0 0}, upload=13160]: Segment reached optimal threshold of 13 pieces. Aug 7 22:27:45.212404510 (12h48m6.956461359s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: context canceled; context canceled Aug 7 22:27:45.212444310 (12h48m6.956501156s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection Aug 7 22:27:45.212507508 (12h48m6.956564355s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection Aug 7 22:27:45.212525428 (12h48m6.956582288s) [seg_pos={0 0}, upload=13160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:27:45.212587349 (12h48m6.956644209s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:27:45.212591472 (12h48m6.956648332s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:45.467451329 (12h48m7.21150818s) [upload=13160]: Done uploading segments... Aug 7 22:27:45.467470231 (12h48m7.211527081s) [upload=13160]: Upload finished. err: <nil> Aug 7 22:28:46.300540067 (12h49m8.044596928s) [upload=13178]: Starting upload Aug 7 22:28:46.300629117 (12h49m8.044685979s) [upload=13178]: Done starting upload Aug 7 22:28:46.300797752 (12h49m8.044854603s) [upload=13178]: Uploading segments... Aug 7 22:28:46.301190738 (12h49m8.045247598s) [upload=13178]: Got next segment. Inline: false Aug 7 22:28:46.301268000 (12h49m8.04532486s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:46.301275318 (12h49m8.045332178s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:29:16.009299446 (12h49m37.753356315s) [upload=13178]: Next segment err: context deadline exceeded Aug 7 22:29:16.009371909 (12h49m37.753428759s) [upload=13178]: Done uploading segments... Aug 7 22:29:16.009392178 (12h49m37.753449035s) [upload=13178]: Upload finished. err: context deadline exceeded Duration: 2.759947942s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...............................X....X............X....] duration: 2.782525099s FAIL (2025-08-07 23:10:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== 5532531s) [upload=13859]: Uploading segments... Aug 7 23:07:36.331975469 (13h27m58.076032329s) [upload=13859]: Got next segment. Inline: false Aug 7 23:07:36.332055014 (13h27m58.076111882s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:36.332082183 (13h27m58.076139048s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:07:36.527006497 (13h27m58.271063364s) [seg_pos={0 0}, upload=13859]: Begin upload segment... Aug 7 23:07:36.527200990 (13h27m58.271257856s) [seg_pos={0 0}, upload=13859]: Done begin upload segment. Aug 7 23:07:36.527296045 (13h27m58.271352917s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.527888445 (13h27m58.271945312s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528191862 (13h27m58.27224877s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.528668127 (13h27m58.272724997s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529016517 (13h27m58.273073388s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529025188 (13h27m58.27308206s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529356889 (13h27m58.273413754s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529522504 (13h27m58.27357949s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.529695262 (13h27m58.273752132s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530083720 (13h27m58.27414058s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530096630 (13h27m58.274153497s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530457173 (13h27m58.274514045s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530555260 (13h27m58.274612125s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.530812907 (13h27m58.274869771s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531000759 (13h27m58.275057625s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531476767 (13h27m58.275533637s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.531916085 (13h27m58.275973s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:36.533640905 (13h27m58.277697776s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Uploading piece... Aug 7 23:07:37.128906885 (13h27m58.872963755s) [upload=13859]: Next returned nil segment Aug 7 23:07:37.128927145 (13h27m58.872984015s) [upload=13859]: Waiting for error group managing segments... Aug 7 23:07:37.290047359 (13h27m59.034104211s) [noise=true , address=104.36.82.38:4699 , piece_id=BHMWGLDQFVRZAVG6UTMX4VLHVMSFZK4Z3TTXFGJ27OETBIMMXFDA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.290266836 (13h27m59.034323685s) [noise=true , address=104.36.82.8:4995 , piece_id=6P2DMWOBSCA7O4RN263WAXKA4RNIVHSXJGET6HTN6PIUSEPD5YKQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.299662430 (13h27m59.043719292s) [noise=true , address=104.36.82.43:4855 , piece_id=I36XW3FFXYG4A4DGP2S43KBTXHXCASU2MEFZO5EASTPFEDVDFKHQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.311258093 (13h27m59.055314954s) [noise=true , address=104.36.82.42:4797 , piece_id=KG624GAXEKUJZFMPETTGG2LFIO6HERMN4UWRNZZXSY5K2NJ62HNA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.314674052 (13h27m59.058730903s) [noise=true , address=104.36.82.60:4338 , piece_id=TICVZEO7Y6JO4K4SU5ZKBTNTHB22GCJWRZFA4IDC32FWAZ4EK7HQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.318326208 (13h27m59.062383068s) [noise=true , address=104.36.82.16:4301 , piece_id=ZUTAB4NKVWUL6NUJPBX26DY4V7PAO2ZQZQ6UDJODI4EKUKLMPBGQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.321228496 (13h27m59.06528535s) [noise=true , address=104.36.82.46:4516 , piece_id=DBHLQ4LK2J3BLKPHSIPQ5DTJ3N64BA73ENNAG5DCD5G3ZLCVPBYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.322848595 (13h27m59.066905466s) [noise=true , address=104.36.82.53:4144 , piece_id=PSQWLMDMJ4FDVN3TN7RVJ7TK4RLOPGNPTCNVIMEJCD4VVVA4B5LQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.327744290 (13h27m59.07180116s) [noise=true , address=45.91.116.8:4077 , piece_id=SDBBK2OVGNTPMP6FB6VULLYU7QLENMD6Y6N5PYBMKUWM3QWTQ4KQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.354647622 (13h27m59.098704477s) [noise=true , address=104.36.82.56:4250 , piece_id=F3JUZL3M7VRSTLZ4ELTRENYJBDO7TLBLB6U62VATX3B5BEOXEB2Q, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.365621120 (13h27m59.10967799s) [noise=true , address=104.36.82.23:4456 , piece_id=WC3ZUPPOSKKUGFFCTW7DHBLKWNPZWTVIUP5ZPKJ6YEAMTGRBITYQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.435610646 (13h27m59.179667496s) [noise=true , address=45.91.116.11:4137 , piece_id=O7QHY4WVDAPGKG2VMWLGYBDJQB2CXWUOKAA6ERUEP7TNQYCO44QQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.551320107 (13h27m59.29537696s) [noise=true , address=104.36.82.62:4409 , piece_id=6WEBZWMER5MCJVH7J2OPEFWKS43CYO6BZVYO2FHKX55Y5PK7R6MQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753359697 (13h27m59.497416543s) [noise=true , address=79.127.205.235:4663 , piece_id=2DB3HAYVKNAXSSNCKENGAC6XFINNKBAYXRCMJT2IBDXQZIJZKQEQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: <nil> Aug 7 23:07:37.753379536 (13h27m59.497436382s) [seg_pos={0 0}, upload=13859]: Segment reached optimal threshold of 14 pieces. Aug 7 23:07:37.753563977 (13h27m59.497620824s) [noise=true , address=104.36.82.61:4388 , piece_id=QEECRWRVMGGQSEEXYLYLMFOLKZPUJHEJ4YSZN4T3K2TYQ7O5LQPA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U9vXtmqghW6VWv8bt5VQEzfZmeLxwPJo6np3sdFWc4FBWSbob): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.755986315 (13h27m59.500043176s) [noise=true , address=64.49.13.155:4922 , piece_id=2OR5ZWM3VGJHGPMMLYXFT2IL2WCHG6B4NPJOXNAYAIX2BSPJEFFQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckJ4ZkPjcb4ZiJ2EauhxsyBMNvGWaVeJxvmiFGYhbxiaLT5uy): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756065009 (13h27m59.50012187s) [noise=true , address=64.49.13.27:4957 , piece_id=3JQV5ZDCT52DY2GSJGLE5PPFI4NJ52Y5NKLC4SWBFPHQET4O36EQ, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD): protocol: expected piece hash; context canceled; EOF Aug 7 23:07:37.756293563 (13h27m59.500350414s) [noise=true , address=45.91.116.12:4255 , piece_id=3OORAIDJ55ARO52VDJYRZPVEUN2FMFHIURFJZSTLLX4ORF6RTELA, seg_pos={0 0}, upload=13859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HLDAXqHSY8tkQVcEY1PCW2MRe7AkZ45TWo6oBPALndznDPPRy): protocol: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40546->45.91.116.12:4255: use of closed network connection Aug 7 23:07:37.756307283 (13h27m59.500364132s) [seg_pos={0 0}, upload=13859]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:07:37.756377087 (13h27m59.500433948s) [upload=13859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:07:37.756382588 (13h27m59.500439532s) [upload=13859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:38.794152538 (13h28m0.538209416s) [upload=13859]: Done uploading segments... Aug 7 23:07:38.794178052 (13h28m0.538234915s) [upload=13859]: Upload finished. err: <nil> Duration: 2.782564601s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration: 2.959034126s FAIL (2025-08-07 23:10:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s... Aug 7 22:21:14.636082751 (12h41m36.380139602s) [upload=13066]: Got next segment. Inline: false Aug 7 22:21:14.636136774 (12h41m36.380193634s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:14.636142452 (12h41m36.380199312s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:16.211357551 (12h41m37.955414411s) [seg_pos={0 0}, upload=13066]: Begin upload segment... Aug 7 22:21:16.211426917 (12h41m37.955483784s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214054689 (12h41m37.95811156s) [seg_pos={0 0}, upload=13066]: Done begin upload segment. Aug 7 22:21:16.214143921 (12h41m37.958200791s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214660710 (12h41m37.95871758s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215094607 (12h41m37.959151469s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215554381 (12h41m37.959611251s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215879389 (12h41m37.959936254s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216161781 (12h41m37.960218643s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216489433 (12h41m37.960546284s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216758002 (12h41m37.960814865s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217037043 (12h41m37.961093908s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217313215 (12h41m37.961370077s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217587290 (12h41m37.961644153s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217852877 (12h41m37.961909742s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218138868 (12h41m37.962195731s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218358245 (12h41m37.962415106s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218693596 (12h41m37.962750459s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218971752 (12h41m37.963028615s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.219339785 (12h41m37.963396657s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.820211995 (12h41m38.564268846s) [upload=13066]: Next returned nil segment Aug 7 22:21:16.820729442 (12h41m38.564786312s) [upload=13066]: Waiting for error group managing segments... Aug 7 22:21:16.974534776 (12h41m38.71859164s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:16.995935209 (12h41m38.739992059s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.005410191 (12h41m38.749467042s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.012303228 (12h41m38.756360079s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.047899041 (12h41m38.791955903s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.065632282 (12h41m38.809689142s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.074330593 (12h41m38.818387457s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.205659931 (12h41m38.949716794s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.304344891 (12h41m39.048401752s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.476320837 (12h41m39.220377687s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.498349653 (12h41m39.242406517s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.566667384 (12h41m39.310724234s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.572654381 (12h41m39.316711251s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002510167 (12h41m39.746567013s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002534446 (12h41m39.746591307s) [seg_pos={0 0}, upload=13066]: Segment reached optimal threshold of 14 pieces. Aug 7 22:21:18.002684920 (12h41m39.746741766s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12goRcYW39Qqga4HW1i6cg6VGKJFUxnyTJBe6HRnZiLAU3LonWn): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002753282 (12h41m39.746810128s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002945708 (12h41m39.747002555s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.003059994 (12h41m39.74711684s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection Aug 7 22:21:18.003080731 (12h41m39.747137592s) [seg_pos={0 0}, upload=13066]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:21:18.003298604 (12h41m39.747355464s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:21:18.003304952 (12h41m39.747361812s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:15.962677772 (12h50m37.706734624s) [upload=13066]: Done uploading segments... Aug 7 22:30:15.962709891 (12h50m37.706766753s) [upload=13066]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.959079998s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.XXXX.X....................................................] duration: 3.15515633s FAIL (2025-08-07 23:10:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.155189815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 1.017827855s FAIL (2025-08-07 23:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4 (12h38m8.493437894s) [upload=13019]: Uploading segments... Aug 7 22:17:46.749423663 (12h38m8.493480532s) [upload=13019]: Got next segment. Inline: true Aug 7 22:17:46.749428712 (12h38m8.493485573s) [upload=13019]: Waiting for error group managing segments... Aug 7 22:17:46.749471140 (12h38m8.49352799s) [upload=13019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:46.749476259 (12h38m8.493533119s) [upload=13019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:46.749479952 (12h38m8.493536802s) [upload=13019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:46.471549080 (12h39m8.215605941s) [upload=13019]: Done uploading segments... Aug 7 22:18:46.471571547 (12h39m8.215628407s) [upload=13019]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:18:46.904761993 (12h39m8.648818857s) [upload=13033]: Starting upload Aug 7 22:18:46.904856076 (12h39m8.648912937s) [upload=13033]: Done starting upload Aug 7 22:18:46.904960146 (12h39m8.649017s) [upload=13033]: Uploading segments... Aug 7 22:18:46.905047245 (12h39m8.649104114s) [upload=13033]: Got next segment. Inline: true Aug 7 22:18:46.905056198 (12h39m8.649113053s) [upload=13033]: Waiting for error group managing segments... Aug 7 22:18:46.905116687 (12h39m8.649173538s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:18:46.905122460 (12h39m8.649179309s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:18:46.905126367 (12h39m8.649183281s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:19:46.626010888 (12h40m8.370067739s) [upload=13033]: Done uploading segments... Aug 7 22:19:46.626032620 (12h40m8.370089471s) [upload=13033]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:19:59.643579230 (12h40m21.387636081s) [upload=13050]: Starting upload Aug 7 22:19:59.643668904 (12h40m21.387725771s) [upload=13050]: Done starting upload Aug 7 22:19:59.643785033 (12h40m21.38784188s) [upload=13050]: Uploading segments... Aug 7 22:19:59.643889787 (12h40m21.387946639s) [upload=13050]: Got next segment. Inline: true Aug 7 22:19:59.643897379 (12h40m21.387954229s) [upload=13050]: Waiting for error group managing segments... Aug 7 22:19:59.643957450 (12h40m21.388014313s) [upload=13050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:59.643962932 (12h40m21.388019793s) [upload=13050]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:59.643966291 (12h40m21.388023151s) [upload=13050]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:59.369875577 (12h41m21.113932442s) [upload=13050]: Done uploading segments... Aug 7 22:20:59.369905671 (12h41m21.113962518s) [upload=13050]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:59.787948456 (12h41m21.532005324s) [upload=13064]: Starting upload Aug 7 22:20:59.788068842 (12h41m21.53212571s) [upload=13064]: Done starting upload Aug 7 22:20:59.788168587 (12h41m21.532225457s) [upload=13064]: Uploading segments... Aug 7 22:20:59.788311286 (12h41m21.532368151s) [upload=13064]: Got next segment. Inline: true Aug 7 22:20:59.788318759 (12h41m21.532375633s) [upload=13064]: Waiting for error group managing segments... Aug 7 22:20:59.788388410 (12h41m21.532445276s) [upload=13064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:59.788394823 (12h41m21.532451689s) [upload=13064]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:59.788398689 (12h41m21.532455554s) [upload=13064]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:59.519361969 (12h42m21.263418825s) [upload=13064]: Done uploading segments... Aug 7 22:21:59.519387607 (12h42m21.263444468s) [upload=13064]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:59.945997169 (12h42m21.69005402s) [upload=13075]: Starting upload Aug 7 22:21:59.946071844 (12h42m21.690129359s) [upload=13075]: Done starting upload Aug 7 22:21:59.946178905 (12h42m21.690235754s) [upload=13075]: Uploading segments... Aug 7 22:21:59.946230067 (12h42m21.690286936s) [upload=13075]: Got next segment. Inline: true Aug 7 22:21:59.946235138 (12h42m21.690292002s) [upload=13075]: Waiting for error group managing segments... Aug 7 22:21:59.946295164 (12h42m21.690352024s) [upload=13075]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:59.946300535 (12h42m21.690357396s) [upload=13075]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:59.946303920 (12h42m21.69036078s) [upload=13075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.677780729 (12h43m21.421837589s) [upload=13075]: Done uploading segments... Aug 7 22:22:59.677798007 (12h43m21.421854868s) [upload=13075]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:00.094834123 (12h43m21.838890994s) [upload=13088]: Starting upload Aug 7 22:23:00.094984258 (12h43m21.83904116s) [upload=13088]: Done starting upload Aug 7 22:23:00.095111897 (12h43m21.839168758s) [upload=13088]: Uploading segments... Aug 7 22:23:00.095477827 (12h43m21.839534697s) [upload=13088]: Got next segment. Inline: true Aug 7 22:23:00.095487112 (12h43m21.839543982s) [upload=13088]: Waiting for error group managing segments... Aug 7 22:23:00.095559737 (12h43m21.839616605s) [upload=13088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:00.095567952 (12h43m21.839624822s) [upload=13088]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:00.095571963 (12h43m21.839628824s) [upload=13088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:59.816074411 (12h44m21.560131259s) [upload=13088]: Done uploading segments... Aug 7 22:23:59.816094923 (12h44m21.560162185s) [upload=13088]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:00.243449247 (12h44m21.987506098s) [upload=13101]: Starting upload Aug 7 22:24:00.243529153 (12h44m21.987586015s) [upload=13101]: Done starting upload Aug 7 22:24:00.243603428 (12h44m21.987660281s) [upload=13101]: Uploading segments... Aug 7 22:24:00.243664741 (12h44m21.98772161s) [upload=13101]: Got next segment. Inline: true Aug 7 22:24:00.243670702 (12h44m21.987727562s) [upload=13101]: Waiting for error group managing segments... Aug 7 22:24:00.243730265 (12h44m21.987787124s) [upload=13101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:00.243736084 (12h44m21.987792946s) [upload=13101]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:00.243739669 (12h44m21.987796531s) [upload=13101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:59.965587620 (12h45m21.709644469s) [upload=13101]: Done uploading segments... Aug 7 22:24:59.965609247 (12h45m21.709666095s) [upload=13101]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:00.417995504 (12h45m22.162052369s) [upload=13115]: Starting upload Aug 7 22:25:00.418148792 (12h45m22.162205658s) [upload=13115]: Done starting upload Aug 7 22:25:00.418313994 (12h45m22.162370855s) [upload=13115]: Uploading segments... Aug 7 22:25:00.418427015 (12h45m22.162483886s) [upload=13115]: Got next segment. Inline: true Aug 7 22:25:00.418453990 (12h45m22.162510859s) [upload=13115]: Waiting for error group managing segments... Aug 7 22:25:00.418542094 (12h45m22.16259896s) [upload=13115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:00.418549113 (12h45m22.162605982s) [upload=13115]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:00.418555718 (12h45m22.162612588s) [upload=13115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:00.143481421 (12h46m21.887538319s) [upload=13115]: Done uploading segments... Aug 7 22:26:00.143508788 (12h46m21.88756568s) [upload=13115]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:00.572183011 (12h46m22.316239858s) [upload=13133]: Starting upload Aug 7 22:26:00.572257919 (12h46m22.316314778s) [upload=13133]: Done starting upload Aug 7 22:26:00.572347104 (12h46m22.316403949s) [upload=13133]: Uploading segments... Aug 7 22:26:00.572414569 (12h46m22.316471429s) [upload=13133]: Got next segment. Inline: true Aug 7 22:26:00.572419354 (12h46m22.316476214s) [upload=13133]: Waiting for error group managing segments... Aug 7 22:26:00.572457012 (12h46m22.316513858s) [upload=13133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:00.572461785 (12h46m22.316518631s) [upload=13133]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:00.572465017 (12h46m22.316521878s) [upload=13133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:00.300711265 (12h47m22.044768125s) [upload=13133]: Done uploading segments... Aug 7 22:27:00.300733422 (12h47m22.044790282s) [upload=13133]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:00.736249379 (12h47m22.48030624s) [upload=13147]: Starting upload Aug 7 22:27:00.736365836 (12h47m22.480422697s) [upload=13147]: Done starting upload Aug 7 22:27:00.736456413 (12h47m22.480513264s) [upload=13147]: Uploading segments... Aug 7 22:27:00.736517869 (12h47m22.480574739s) [upload=13147]: Got next segment. Inline: true Aug 7 22:27:00.736526521 (12h47m22.480583386s) [upload=13147]: Waiting for error group managing segments... Aug 7 22:27:00.736600220 (12h47m22.480657069s) [upload=13147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:00.736605984 (12h47m22.480662844s) [upload=13147]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:00.736609354 (12h47m22.480666214s) [upload=13147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:00.459012227 (12h48m22.203069105s) [upload=13147]: Done uploading segments... Aug 7 22:28:00.459039460 (12h48m22.203096312s) [upload=13147]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.017888303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X.X.XX.....................................................] duration: 895.878767ms FAIL (2025-08-07 23:10:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 895.9134ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.570880398s FAIL (2025-08-07 23:09:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.57091104s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXX..X..................................................] duration: 2.593311052s FAIL (2025-08-07 23:09:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7 22:27:44.139472223 (12h48m5.883529083s) [upload=13160]: Got next segment. Inline: false Aug 7 22:27:44.139508151 (12h48m5.883565011s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:44.139513399 (12h48m5.883570259s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:27:44.245907064 (12h48m5.989963927s) [seg_pos={0 0}, upload=13160]: Begin upload segment... Aug 7 22:27:44.246039735 (12h48m5.990096597s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.246596012 (12h48m5.990652882s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247006251 (12h48m5.991063115s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247197178 (12h48m5.991254041s) [seg_pos={0 0}, upload=13160]: Done begin upload segment. Aug 7 22:27:44.247225152 (12h48m5.991282018s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247751444 (12h48m5.99180831s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.247956514 (12h48m5.992013382s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248499391 (12h48m5.992556252s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.248606386 (12h48m5.992663249s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249110151 (12h48m5.99316702s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249195913 (12h48m5.993252781s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249587895 (12h48m5.993644759s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.249756005 (12h48m5.993812872s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.250801847 (12h48m5.99485871s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251094602 (12h48m5.995151463s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251408354 (12h48m5.995465217s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.251693979 (12h48m5.995750839s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Uploading piece... Aug 7 22:27:44.928589281 (12h48m6.672646155s) [upload=13160]: Next returned nil segment Aug 7 22:27:44.928610126 (12h48m6.672666995s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:27:45.055730069 (12h48m6.799786923s) [noise=true , address=104.36.82.11:4115 , piece_id=B2C72IS3FGY7DT4SXLUG5FYHCN5ECAMCAIVT2ZNVCXML2RVCQZFQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.058230571 (12h48m6.802287421s) [noise=true , address=104.36.82.4:4801 , piece_id=U6XCDALXGOTJHRL2P4V7DOPIJJF6VUKB5RIJYOP3WXFTC6HQKQXQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.072176193 (12h48m6.816233043s) [noise=true , address=104.36.82.6:4876 , piece_id=45DZFNAUCX6EJNPT4WBVLEIO2GZROYIF2NQ254LU32T6AE7P4ZRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.076144433 (12h48m6.82020128s) [noise=true , address=104.36.82.5:4837 , piece_id=XDA3KTWSH5OJPHK4IFOPINMJAR6VJLL7B7WNO3MV2VT3HQCGJ2KA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.083969611 (12h48m6.828026457s) [noise=true , address=104.36.82.13:4185 , piece_id=QI54Y53UTBR7JEQ4NPD5R37EC47364LEUWETTCGI3WQUKSHUTP5A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.090499988 (12h48m6.834556849s) [noise=true , address=104.36.82.10:4078 , piece_id=UCDTQMJTYXAMJXGRPKBGTJZ4OCF4P2BSCR6WR3KSXCK6S5PWWZ7A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.129675895 (12h48m6.873732757s) [noise=true , address=45.91.116.12:4225 , piece_id=N7R42AWZKTHBBPZY24OVKSBRTPGVBLJ3HBYB2ONRZ56F6YZK2XWQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.148504731 (12h48m6.892561581s) [noise=true , address=77.90.31.10:4517 , piece_id=QHV63D3QBJ5GVDWNHGWJTLXQNHGK4PZ4OCT4UWCEQKHMYBDL26WA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.183435281 (12h48m6.927492128s) [noise=true , address=104.36.82.10:4079 , piece_id=2ADIUNAB5FPKLXMW46HBB75SJLYOKA4N3TYS6J72WZDKLQTSP23A, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.188056276 (12h48m6.932113137s) [noise=true , address=77.90.31.12:4590 , piece_id=IYZTGAPBVXXQNQVDNR4NZRPDNNIBETBQ26DZUZ2SODA2D2VQUXTA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.189109860 (12h48m6.93316672s) [noise=true , address=77.90.31.13:4628 , piece_id=FSARBSWO4VFKOYHINODNYG7BC4JEH6KX5Z3IOYIJ4DGQ3DCZNDJA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.191395584 (12h48m6.93545243s) [noise=true , address=45.91.116.9:4261 , piece_id=ZESRB2XAQ4WPIB2HQKYLZKJDXDE65NXMRX4M5JE3EKRHROCT4JKA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212167167 (12h48m6.956224014s) [noise=true , address=77.90.31.11:4553 , piece_id=ZDJ5BW2XPNHEBNB6FBVWAH66T6SVKAUY3724YY6HZ25VBLKYFTIQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: <nil> Aug 7 22:27:45.212191124 (12h48m6.956247983s) [seg_pos={0 0}, upload=13160]: Segment reached optimal threshold of 13 pieces. Aug 7 22:27:45.212404510 (12h48m6.956461359s) [noise=true , address=45.91.116.8:4047 , piece_id=LDURVPEWVV7OEPVY4L4ZER4H2EDUPJS45N55Y57VZUTLSTKGXSRA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: context canceled; context canceled Aug 7 22:27:45.212444310 (12h48m6.956501156s) [noise=true , address=77.90.31.18:4806 , piece_id=ZJLVNSXETRBXYNHIDLN7TSQU6OQVWTCOQFJLPWAMCPY2JHBTHVQA, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44926->77.90.31.18:4806: use of closed network connection Aug 7 22:27:45.212507508 (12h48m6.956564355s) [noise=true , address=45.91.116.5:4297 , piece_id=QEQZU67YDQXNW7ACIRVXJKX2HIKMZMAP23TCAMTB2RDK3RFKLLRQ, seg_pos={0 0}, upload=13160]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41424->45.91.116.5:4297: use of closed network connection Aug 7 22:27:45.212525428 (12h48m6.956582288s) [seg_pos={0 0}, upload=13160]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:27:45.212587349 (12h48m6.956644209s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:27:45.212591472 (12h48m6.956648332s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:45.467451329 (12h48m7.21150818s) [upload=13160]: Done uploading segments... Aug 7 22:27:45.467470231 (12h48m7.211527081s) [upload=13160]: Upload finished. err: <nil> Aug 7 22:28:46.300540067 (12h49m8.044596928s) [upload=13178]: Starting upload Aug 7 22:28:46.300629117 (12h49m8.044685979s) [upload=13178]: Done starting upload Aug 7 22:28:46.300797752 (12h49m8.044854603s) [upload=13178]: Uploading segments... Aug 7 22:28:46.301190738 (12h49m8.045247598s) [upload=13178]: Got next segment. Inline: false Aug 7 22:28:46.301268000 (12h49m8.04532486s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:46.301275318 (12h49m8.045332178s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:29:16.009299446 (12h49m37.753356315s) [upload=13178]: Next segment err: context deadline exceeded Aug 7 22:29:16.009371909 (12h49m37.753428759s) [upload=13178]: Done uploading segments... Aug 7 22:29:16.009392178 (12h49m37.753449035s) [upload=13178]: Upload finished. err: context deadline exceeded Duration: 2.593338922s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.096318224s FAIL (2025-08-07 23:09:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.096357485s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.149102104s FAIL (2025-08-07 23:09:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.149131274s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XXXXXXXX...................................................] duration: 3.275428172s FAIL (2025-08-07 23:09:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s... Aug 7 22:21:14.636082751 (12h41m36.380139602s) [upload=13066]: Got next segment. Inline: false Aug 7 22:21:14.636136774 (12h41m36.380193634s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:14.636142452 (12h41m36.380199312s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:21:16.211357551 (12h41m37.955414411s) [seg_pos={0 0}, upload=13066]: Begin upload segment... Aug 7 22:21:16.211426917 (12h41m37.955483784s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214054689 (12h41m37.95811156s) [seg_pos={0 0}, upload=13066]: Done begin upload segment. Aug 7 22:21:16.214143921 (12h41m37.958200791s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.214660710 (12h41m37.95871758s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215094607 (12h41m37.959151469s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215554381 (12h41m37.959611251s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.215879389 (12h41m37.959936254s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216161781 (12h41m37.960218643s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216489433 (12h41m37.960546284s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.216758002 (12h41m37.960814865s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217037043 (12h41m37.961093908s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217313215 (12h41m37.961370077s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217587290 (12h41m37.961644153s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.217852877 (12h41m37.961909742s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218138868 (12h41m37.962195731s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218358245 (12h41m37.962415106s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218693596 (12h41m37.962750459s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.218971752 (12h41m37.963028615s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.219339785 (12h41m37.963396657s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Uploading piece... Aug 7 22:21:16.820211995 (12h41m38.564268846s) [upload=13066]: Next returned nil segment Aug 7 22:21:16.820729442 (12h41m38.564786312s) [upload=13066]: Waiting for error group managing segments... Aug 7 22:21:16.974534776 (12h41m38.71859164s) [noise=true , address=104.36.82.61:4397 , piece_id=7FJT5CLASEMVAP4HSGHP3XQTNBKVRP2STW2YF6H2UDLHFPUHKMMQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:16.995935209 (12h41m38.739992059s) [noise=true , address=104.36.82.9:4036 , piece_id=UG2DKHGMS5NC3BTN7QQU23JXBUQ7UFVN6FZKPBZHYUOQ6SEJV5WA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.005410191 (12h41m38.749467042s) [noise=true , address=104.36.82.60:4354 , piece_id=A3YTMCWULXHPBRVL6TDQOBNYTQQWNKB2UZPKAPDH2CCG5SMV6QTA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.012303228 (12h41m38.756360079s) [noise=true , address=104.36.82.17:4344 , piece_id=CCKNSMXAJAO3QUJDDZ7POWA7NZ63KACQQAJRU37RPG5YMXVMOVZA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.047899041 (12h41m38.791955903s) [noise=true , address=104.36.82.13:4191 , piece_id=NUNWAQKF465J5JQLYQBOCH75SAXPR5F3OPBDDTC6AWWBSKB6MSUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.065632282 (12h41m38.809689142s) [noise=true , address=104.36.82.60:4339 , piece_id=FI3A54HMEOKZJ2FC2ZOLNQXW6NR7AUD26HBVFFGEGTG3AYRZD73A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.074330593 (12h41m38.818387457s) [noise=true , address=104.36.82.62:4435 , piece_id=O3PP5IRA3MLOLM35Z2U2BLM7D6GPH6WMHWANUF7AAGTW7IW6CZBQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.205659931 (12h41m38.949716794s) [noise=true , address=104.36.82.59:4336 , piece_id=5AVF5EEMC334PPHVOSYDAYZ5CKQ4F5LE6BRD5G446NRRJNYQBOYQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.304344891 (12h41m39.048401752s) [noise=true , address=45.91.116.5:4327 , piece_id=IGBMWNLDFIKQY7AJ3UJ6JILAFLRQ5DZAE6FFO3EJ4NDPVWZXCK5A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.476320837 (12h41m39.220377687s) [noise=true , address=64.49.13.140:4509 , piece_id=CNKDQOVU53HJ6EDOSZOXRSFOKQBNKWERZSFJQA5F24FZBR6XEEQA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.498349653 (12h41m39.242406517s) [noise=true , address=64.49.13.26:4956 , piece_id=VHWLGKKB5MBWQOD3KVNXE3YYCK5TZ56UU25BTACIPJQZJA7PHA7Q, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.566667384 (12h41m39.310724234s) [noise=true , address=104.36.82.47:4525 , piece_id=62GQ6IOY4OBZAPRBVZ2OLLA7R4DVQVMWNQZAR67BTPQUX2HUEZRA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:17.572654381 (12h41m39.316711251s) [noise=true , address=79.127.205.233:4647 , piece_id=6YIT5HDQJ6CN63ZFANACICFS4M2OQEPOSFIV36QXS5VLJZ2BRR7A, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002510167 (12h41m39.746567013s) [noise=true , address=45.91.116.9:4292 , piece_id=NO3S4IIMN5JWUNMMNQS6SDFLCDPW3MF3LK4BDZ4IVIZMRWBHSTQQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: <nil> Aug 7 22:21:18.002534446 (12h41m39.746591307s) [seg_pos={0 0}, upload=13066]: Segment reached optimal threshold of 14 pieces. Aug 7 22:21:18.002684920 (12h41m39.746741766s) [noise=true , address=104.36.82.43:4849 , piece_id=ECUDD3LRMPQMXZW6MRFLFHVPAO3RHCV6MWZYOJSASL4ILT54TFFA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12goRcYW39Qqga4HW1i6cg6VGKJFUxnyTJBe6HRnZiLAU3LonWn): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002753282 (12h41m39.746810128s) [noise=true , address=104.36.82.58:4270 , piece_id=LUZ6HPESCVZDYNDVHR2QIS6HM2IIRK3IQB6BSV2VVVB3OFRZVMFQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.002945708 (12h41m39.747002555s) [noise=true , address=104.36.82.2:4754 , piece_id=PQJBMG6FQ3HUTE26IBB5R3WQCAJTFKMGECOXSJPV2VDHP33JOPUA, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs): protocol: expected piece hash; context canceled; EOF Aug 7 22:21:18.003059994 (12h41m39.74711684s) [noise=true , address=104.36.82.44:4904 , piece_id=RWD3KVEEHT5BUVYBAHRHZFCBTCZUTNHRD6PU7OWOUIF2QVBZXSSQ, seg_pos={0 0}, upload=13066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47438->104.36.82.44:4904: use of closed network connection Aug 7 22:21:18.003080731 (12h41m39.747137592s) [seg_pos={0 0}, upload=13066]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:21:18.003298604 (12h41m39.747355464s) [upload=13066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:21:18.003304952 (12h41m39.747361812s) [upload=13066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:15.962677772 (12h50m37.706734624s) [upload=13066]: Done uploading segments... Aug 7 22:30:15.962709891 (12h50m37.706766753s) [upload=13066]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.275458691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........X............................................] duration: 956.570775ms FAIL (2025-08-07 23:09:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7a9bb586841bcbba387104cc5acb2fb0226d7912ebd238eb0e2b5d0ce19a4290 != before 385757229355cc02ac87a6a03c913b16ce6a2b0803d9178de6d55aa5b1a5df0f ======LOGS====== .906985333s) [upload=13211]: Done uploading segments... Aug 7 22:30:48.162948603 (12h51m9.907005457s) [upload=13211]: Upload finished. err: <nil> Aug 7 22:31:41.459072464 (12h52m3.203129325s) [upload=13225]: Starting upload Aug 7 22:31:41.459235349 (12h52m3.20329221s) [upload=13225]: Done starting upload Aug 7 22:31:41.459324787 (12h52m3.203381637s) [upload=13225]: Uploading segments... Aug 7 22:31:41.459411951 (12h52m3.20346882s) [upload=13225]: Got next segment. Inline: true Aug 7 22:31:41.459430146 (12h52m3.203487009s) [upload=13225]: Waiting for error group managing segments... Aug 7 22:31:41.459488485 (12h52m3.203545341s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:41.459506915 (12h52m3.203563776s) [upload=13225]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:31:41.459510588 (12h52m3.203567448s) [upload=13225]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:31:41.614411609 (12h52m3.358468456s) [upload=13225]: Done uploading segments... Aug 7 22:31:41.614450157 (12h52m3.358507003s) [upload=13225]: Upload finished. err: <nil> Aug 7 22:32:40.463985578 (12h53m2.208042449s) [upload=13243]: Starting upload Aug 7 22:32:40.464053463 (12h53m2.208110332s) [upload=13243]: Done starting upload Aug 7 22:32:40.464142006 (12h53m2.208198871s) [upload=13243]: Uploading segments... Aug 7 22:32:40.464193502 (12h53m2.208250372s) [upload=13243]: Got next segment. Inline: true Aug 7 22:32:40.464200462 (12h53m2.208257333s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:32:40.464254018 (12h53m2.208310881s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:32:40.464260315 (12h53m2.208317186s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:32:40.464264348 (12h53m2.208321216s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:32:40.626697153 (12h53m2.370754019s) [upload=13243]: Done uploading segments... Aug 7 22:32:40.626740512 (12h53m2.370797373s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:33:27.614940485 (12h53m49.358997337s) [upload=13259]: Starting upload Aug 7 22:33:27.615010564 (12h53m49.359067437s) [upload=13259]: Done starting upload Aug 7 22:33:27.615264115 (12h53m49.35932096s) [upload=13259]: Uploading segments... Aug 7 22:33:27.615348974 (12h53m49.359405844s) [upload=13259]: Got next segment. Inline: true Aug 7 22:33:27.615369687 (12h53m49.359426557s) [upload=13259]: Waiting for error group managing segments... Aug 7 22:33:27.615423331 (12h53m49.359480181s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:27.615441056 (12h53m49.359497916s) [upload=13259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:27.615444764 (12h53m49.359501624s) [upload=13259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:27.920082993 (12h53m49.664139864s) [upload=13259]: Done uploading segments... Aug 7 22:33:27.920126071 (12h53m49.664182944s) [upload=13259]: Upload finished. err: <nil> Aug 7 22:33:46.543534425 (12h54m8.28759129s) [upload=13262]: Starting upload Aug 7 22:33:46.543613309 (12h54m8.287670173s) [upload=13262]: Done starting upload Aug 7 22:33:46.543768120 (12h54m8.287824974s) [upload=13262]: Uploading segments... Aug 7 22:33:46.543821818 (12h54m8.287896467s) [upload=13262]: Got next segment. Inline: true Aug 7 22:33:46.543846433 (12h54m8.287903298s) [upload=13262]: Waiting for error group managing segments... Aug 7 22:33:46.543911325 (12h54m8.287968186s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:33:46.543917553 (12h54m8.287974414s) [upload=13262]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:33:46.543922581 (12h54m8.287979442s) [upload=13262]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:46.689591840 (12h54m8.43364869s) [upload=13262]: Done uploading segments... Aug 7 22:33:46.689611228 (12h54m8.433668079s) [upload=13262]: Upload finished. err: <nil> Aug 7 22:34:17.900718443 (12h54m39.644775303s) [upload=13273]: Starting upload Aug 7 22:34:17.900776243 (12h54m39.644833096s) [upload=13273]: Done starting upload Aug 7 22:34:17.900871893 (12h54m39.644928745s) [upload=13273]: Uploading segments... Aug 7 22:34:17.900917644 (12h54m39.644974513s) [upload=13273]: Got next segment. Inline: true Aug 7 22:34:17.900922943 (12h54m39.644979803s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:34:17.900967950 (12h54m39.645024799s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:17.900973539 (12h54m39.645030389s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:34:17.900977174 (12h54m39.645034035s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:34:18.229545674 (12h54m39.973602525s) [upload=13273]: Done uploading segments... Aug 7 22:34:18.229564006 (12h54m39.973620856s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:35:15.773724797 (12h55m37.517781661s) [upload=13291]: Starting upload Aug 7 22:35:15.773809791 (12h55m37.517866659s) [upload=13291]: Done starting upload Aug 7 22:35:15.773932695 (12h55m37.517989544s) [upload=13291]: Uploading segments... Aug 7 22:35:15.774007738 (12h55m37.518064606s) [upload=13291]: Got next segment. Inline: true Aug 7 22:35:15.774013953 (12h55m37.518070813s) [upload=13291]: Waiting for error group managing segments... Aug 7 22:35:15.774065151 (12h55m37.518122002s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:15.774072234 (12h55m37.518129094s) [upload=13291]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:15.774075519 (12h55m37.518132379s) [upload=13291]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:15.991444925 (12h55m37.735501779s) [upload=13291]: Done uploading segments... Aug 7 22:35:15.991477669 (12h55m37.735534518s) [upload=13291]: Upload finished. err: <nil> Aug 7 22:35:45.697797956 (12h56m7.441854806s) [upload=13299]: Starting upload Aug 7 22:35:45.697851126 (12h56m7.441907986s) [upload=13299]: Done starting upload Aug 7 22:35:45.697942841 (12h56m7.441999692s) [upload=13299]: Uploading segments... Aug 7 22:35:45.697986029 (12h56m7.442042898s) [upload=13299]: Got next segment. Inline: true Aug 7 22:35:45.697991340 (12h56m7.442048201s) [upload=13299]: Waiting for error group managing segments... Aug 7 22:35:45.698035363 (12h56m7.442092214s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:35:45.698040424 (12h56m7.442097274s) [upload=13299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:35:45.698043821 (12h56m7.442100673s) [upload=13299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:35:46.104801775 (12h56m7.848858636s) [upload=13299]: Done uploading segments... Aug 7 22:35:46.104833054 (12h56m7.848889914s) [upload=13299]: Upload finished. err: <nil> Aug 7 22:36:15.294117977 (12h56m37.038174844s) [upload=13312]: Starting upload Aug 7 22:36:15.294188736 (12h56m37.038245585s) [upload=13312]: Done starting upload Aug 7 22:36:15.294275452 (12h56m37.038332302s) [upload=13312]: Uploading segments... Aug 7 22:36:15.294329447 (12h56m37.038386319s) [upload=13312]: Got next segment. Inline: true Aug 7 22:36:15.294336861 (12h56m37.038393722s) [upload=13312]: Waiting for error group managing segments... Aug 7 22:36:15.294392063 (12h56m37.038448923s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:36:15.294399087 (12h56m37.03845605s) [upload=13312]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:36:15.294403113 (12h56m37.03845998s) [upload=13312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:36:15.516924569 (12h56m37.260981431s) [upload=13312]: Done uploading segments... Aug 7 22:36:15.516963723 (12h56m37.261020572s) [upload=13312]: Upload finished. err: <nil> Aug 7 22:37:19.874350488 (12h57m41.618407338s) [upload=13332]: Starting upload Aug 7 22:37:19.874402146 (12h57m41.618459014s) [upload=13332]: Done starting upload Aug 7 22:37:19.874509826 (12h57m41.618566677s) [upload=13332]: Uploading segments... Aug 7 22:37:19.874550139 (12h57m41.618607008s) [upload=13332]: Got next segment. Inline: true Aug 7 22:37:19.874555598 (12h57m41.618612458s) [upload=13332]: Waiting for error group managing segments... Aug 7 22:37:19.874613141 (12h57m41.618669991s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:19.874620379 (12h57m41.618677239s) [upload=13332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:19.874623922 (12h57m41.618680782s) [upload=13332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:20.019547240 (12h57m41.763604101s) [upload=13332]: Done uploading segments... Aug 7 22:37:20.019586845 (12h57m41.763643695s) [upload=13332]: Upload finished. err: <nil> Aug 7 22:37:52.930214271 (12h58m14.674271118s) [upload=13342]: Starting upload Aug 7 22:37:52.930276626 (12h58m14.674333472s) [upload=13342]: Done starting upload Aug 7 22:37:52.930355150 (12h58m14.674411996s) [upload=13342]: Uploading segments... Aug 7 22:37:52.930403037 (12h58m14.674459908s) [upload=13342]: Got next segment. Inline: true Aug 7 22:37:52.930408052 (12h58m14.674464912s) [upload=13342]: Waiting for error group managing segments... Aug 7 22:37:52.930446802 (12h58m14.674503648s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:37:52.930451272 (12h58m14.674508132s) [upload=13342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:37:52.930464087 (12h58m14.674520947s) [upload=13342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:37:53.096265259 (12h58m14.840322111s) [upload=13342]: Done uploading segments... Aug 7 22:37:53.096285906 (12h58m14.840342756s) [upload=13342]: Upload finished. err: <nil> Duration: 956.606429ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.197514852s FAIL (2025-08-07 23:09:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.197554894s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [XX....X.....................................................] duration: 939.037044ms FAIL (2025-08-07 23:09:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=13077]: Starting upload Aug 7 22:22:03.530157956 (12h42m25.27421483s) [upload=13077]: Done starting upload Aug 7 22:22:03.530254340 (12h42m25.274311199s) [upload=13077]: Uploading segments... Aug 7 22:22:03.530319521 (12h42m25.274376398s) [upload=13077]: Got next segment. Inline: true Aug 7 22:22:03.530326839 (12h42m25.27438371s) [upload=13077]: Waiting for error group managing segments... Aug 7 22:22:03.530387033 (12h42m25.274443903s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:03.530393764 (12h42m25.274450635s) [upload=13077]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:03.530398068 (12h42m25.274454934s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:03.668101987 (12h42m25.412158856s) [upload=13077]: Done uploading segments... Aug 7 22:22:03.668146181 (12h42m25.412203031s) [upload=13077]: Upload finished. err: <nil> Aug 7 22:22:50.781175997 (12h43m12.525232844s) [upload=13085]: Starting upload Aug 7 22:22:50.781238492 (12h43m12.525295338s) [upload=13085]: Done starting upload Aug 7 22:22:50.781295832 (12h43m12.525352678s) [upload=13085]: Uploading segments... Aug 7 22:22:50.781339974 (12h43m12.525396844s) [upload=13085]: Got next segment. Inline: true Aug 7 22:22:50.781344619 (12h43m12.525401479s) [upload=13085]: Waiting for error group managing segments... Aug 7 22:22:50.781382529 (12h43m12.525439375s) [upload=13085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:50.781387267 (12h43m12.525444113s) [upload=13085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:50.781389841 (12h43m12.525446704s) [upload=13085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:20.513298723 (12h43m42.257355587s) [upload=13085]: Done uploading segments... Aug 7 22:23:20.513325654 (12h43m42.25738252s) [upload=13085]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:51.254516692 (12h44m12.998573555s) [upload=13099]: Starting upload Aug 7 22:23:51.254606307 (12h44m12.998663169s) [upload=13099]: Done starting upload Aug 7 22:23:51.254669521 (12h44m12.998726371s) [upload=13099]: Uploading segments... Aug 7 22:23:51.254709863 (12h44m12.998766733s) [upload=13099]: Got next segment. Inline: true Aug 7 22:23:51.254715030 (12h44m12.99877189s) [upload=13099]: Waiting for error group managing segments... Aug 7 22:23:51.254768832 (12h44m12.998825693s) [upload=13099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:51.254774395 (12h44m12.998831255s) [upload=13099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:51.254778260 (12h44m12.998835121s) [upload=13099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:20.980586635 (12h44m42.724643486s) [upload=13099]: Done uploading segments... Aug 7 22:24:20.980611487 (12h44m42.724668337s) [upload=13099]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:22.809871679 (12h44m44.55392853s) [upload=13108]: Starting upload Aug 7 22:24:22.809959685 (12h44m44.554016545s) [upload=13108]: Done starting upload Aug 7 22:24:22.810026016 (12h44m44.55408288s) [upload=13108]: Uploading segments... Aug 7 22:24:22.810084984 (12h44m44.554141856s) [upload=13108]: Got next segment. Inline: true Aug 7 22:24:22.810090937 (12h44m44.554147807s) [upload=13108]: Waiting for error group managing segments... Aug 7 22:24:22.810180251 (12h44m44.554237121s) [upload=13108]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:22.810187976 (12h44m44.554244835s) [upload=13108]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:22.810192661 (12h44m44.55424953s) [upload=13108]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:52.535275009 (12h45m14.279331864s) [upload=13108]: Done uploading segments... Aug 7 22:24:52.535303649 (12h45m14.279360515s) [upload=13108]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:52.960601883 (12h45m14.70465875s) [upload=13112]: Starting upload Aug 7 22:24:52.960708121 (12h45m14.704764984s) [upload=13112]: Done starting upload Aug 7 22:24:52.960845573 (12h45m14.704902419s) [upload=13112]: Uploading segments... Aug 7 22:24:52.960931513 (12h45m14.704988384s) [upload=13112]: Got next segment. Inline: true Aug 7 22:24:52.960939376 (12h45m14.704996243s) [upload=13112]: Waiting for error group managing segments... Aug 7 22:24:52.961019433 (12h45m14.705076313s) [upload=13112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:52.961026755 (12h45m14.705083622s) [upload=13112]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:52.961030905 (12h45m14.705087769s) [upload=13112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:22.687567107 (12h45m44.431623957s) [upload=13112]: Done uploading segments... Aug 7 22:25:22.687590061 (12h45m44.431646912s) [upload=13112]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:23.131013864 (12h45m44.875070712s) [upload=13122]: Starting upload Aug 7 22:25:23.131088619 (12h45m44.875145482s) [upload=13122]: Done starting upload Aug 7 22:25:23.131275088 (12h45m44.875331941s) [upload=13122]: Uploading segments... Aug 7 22:25:23.131376992 (12h45m44.875433843s) [upload=13122]: Got next segment. Inline: true Aug 7 22:25:23.131382965 (12h45m44.875439816s) [upload=13122]: Waiting for error group managing segments... Aug 7 22:25:23.131434882 (12h45m44.875491733s) [upload=13122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:23.131440120 (12h45m44.875496982s) [upload=13122]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:23.131443668 (12h45m44.875500519s) [upload=13122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:23.272670449 (12h45m45.016727299s) [upload=13122]: Done uploading segments... Aug 7 22:25:23.272702241 (12h45m45.016759091s) [upload=13122]: Upload finished. err: <nil> Aug 7 22:26:25.811198326 (12h46m47.555255193s) [upload=13140]: Starting upload Aug 7 22:26:25.811285415 (12h46m47.555342279s) [upload=13140]: Done starting upload Aug 7 22:26:25.811369732 (12h46m47.555426582s) [upload=13140]: Uploading segments... Aug 7 22:26:25.811467114 (12h46m47.55552398s) [upload=13140]: Got next segment. Inline: true Aug 7 22:26:25.811473701 (12h46m47.555530552s) [upload=13140]: Waiting for error group managing segments... Aug 7 22:26:25.811539126 (12h46m47.555595979s) [upload=13140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:25.811545282 (12h46m47.555602146s) [upload=13140]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:25.811548939 (12h46m47.555605802s) [upload=13140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:25.968679249 (12h46m47.712736113s) [upload=13140]: Done uploading segments... Aug 7 22:26:25.968709074 (12h46m47.712765928s) [upload=13140]: Upload finished. err: <nil> Aug 7 22:27:27.799853559 (12h47m49.543910409s) [upload=13154]: Starting upload Aug 7 22:27:27.799998197 (12h47m49.544055055s) [upload=13154]: Done starting upload Aug 7 22:27:27.800097878 (12h47m49.544154732s) [upload=13154]: Uploading segments... Aug 7 22:27:27.800202613 (12h47m49.544259463s) [upload=13154]: Got next segment. Inline: true Aug 7 22:27:27.800221436 (12h47m49.544278297s) [upload=13154]: Waiting for error group managing segments... Aug 7 22:27:27.800305006 (12h47m49.544361867s) [upload=13154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:27.800310790 (12h47m49.54436765s) [upload=13154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:27.800314678 (12h47m49.544371539s) [upload=13154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:27.950105257 (12h47m49.694162117s) [upload=13154]: Done uploading segments... Aug 7 22:27:27.950124372 (12h47m49.694181233s) [upload=13154]: Upload finished. err: <nil> Aug 7 22:27:59.550898410 (12h48m21.29495526s) [upload=13165]: Starting upload Aug 7 22:27:59.550970829 (12h48m21.295027691s) [upload=13165]: Done starting upload Aug 7 22:27:59.551049380 (12h48m21.295106232s) [upload=13165]: Uploading segments... Aug 7 22:27:59.551283044 (12h48m21.295339906s) [upload=13165]: Got next segment. Inline: true Aug 7 22:27:59.551293209 (12h48m21.295350069s) [upload=13165]: Waiting for error group managing segments... Aug 7 22:27:59.551359587 (12h48m21.295416438s) [upload=13165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:59.551365951 (12h48m21.295422812s) [upload=13165]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:59.551369453 (12h48m21.295426314s) [upload=13165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:59.691525895 (12h48m21.435582778s) [upload=13165]: Done uploading segments... Aug 7 22:27:59.691551086 (12h48m21.435607933s) [upload=13165]: Upload finished. err: <nil> Aug 7 22:29:06.342665932 (12h49m28.086722791s) [upload=13183]: Starting upload Aug 7 22:29:06.342756156 (12h49m28.086813016s) [upload=13183]: Done starting upload Aug 7 22:29:06.342875471 (12h49m28.086932321s) [upload=13183]: Uploading segments... Aug 7 22:29:06.343020027 (12h49m28.087076894s) [upload=13183]: Got next segment. Inline: true Aug 7 22:29:06.343029056 (12h49m28.087085926s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:29:06.343276816 (12h49m28.087333721s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:06.343287054 (12h49m28.087343919s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:06.343290895 (12h49m28.08734776s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:36.070785151 (12h49m57.814842003s) [upload=13183]: Done uploading segments... Aug 7 22:29:36.070807148 (12h49m57.814863999s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Duration: 939.073045ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)