Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XX..XXXX.X...........X.XXXX.XX......XXXXX...XX.X...X......] duration: 21.766322707s FAIL (2025-07-27 00:47:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== :41:07.222615952 (61h5m38.412914523s) [upload=58559]: Got next segment. Inline: false Jul 27 00:41:07.222678612 (61h5m38.41297719s) [upload=58559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:07.222684889 (61h5m38.412983462s) [upload=58559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:41:07.330155865 (61h5m38.520454446s) [seg_pos={0 0}, upload=58559]: Begin upload segment... Jul 27 00:41:07.330223641 (61h5m38.520522225s) [seg_pos={0 0}, upload=58559]: Done begin upload segment. Jul 27 00:41:07.330253409 (61h5m38.520551992s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.331859693 (61h5m38.522158283s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332330366 (61h5m38.52262895s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332739898 (61h5m38.523038466s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333152185 (61h5m38.523450752s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333682612 (61h5m38.523981181s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334071276 (61h5m38.524369844s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334428527 (61h5m38.524727095s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334781140 (61h5m38.52507972s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335157956 (61h5m38.525456544s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335505072 (61h5m38.525803652s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335862982 (61h5m38.526161564s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336223096 (61h5m38.526521679s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336566256 (61h5m38.526864838s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336924464 (61h5m38.527223047s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.337308564 (61h5m38.527607147s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.959715000 (61h5m39.150013589s) [upload=58559]: Next returned nil segment Jul 27 00:41:07.959737017 (61h5m39.150035606s) [upload=58559]: Waiting for error group managing segments... Jul 27 00:41:08.098609518 (61h5m39.288908087s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.114056645 (61h5m39.30435523s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115224636 (61h5m39.305523208s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115378353 (61h5m39.305676945s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.119763396 (61h5m39.310061969s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.131072499 (61h5m39.321371079s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.192043578 (61h5m39.382342152s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:09.562998275 (61h5m40.753296862s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:26.072685752 (61h5m57.262984318s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:36.947344132 (61h6m8.137642702s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947443385 (61h6m8.137741965s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection Jul 27 00:41:36.947487056 (61h6m8.137785647s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947752065 (61h6m8.138050651s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection Jul 27 00:41:36.947790491 (61h6m8.138089061s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection Jul 27 00:41:36.947845062 (61h6m8.138143632s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 139.59.144.22:49972->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:36.947886799 (61h6m8.138185382s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection Jul 27 00:41:36.947923797 (61h6m8.138222386s) [seg_pos={0 0}, upload=58559]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:41:37.055592756 (61h6m8.24589133s) [upload=58559]: Done uploading segments... Jul 27 00:41:37.055626444 (61h6m8.245925032s) [upload=58559]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 21.766360661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.....] duration: 2.481589711s FAIL (2025-07-27 00:47:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.481615694s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX...........] duration: 5.086227664s FAIL (2025-07-27 00:47:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.086266271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X.X..X.X...............................X...................] duration: 10.996896363s FAIL (2025-07-27 00:47:08 UTC) orbiter warning: context deadline exceeded Duration: 10.996923906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX..........] duration: 34.616555999s FAIL (2025-07-27 00:47:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 34.616588123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....X...XX....X......XXXX..XX...X.....X.X....X...........] duration: 14.010326623s FAIL (2025-07-27 00:47:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.010342344s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 856.285208ms FAIL (2025-07-27 00:46:53 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 856.329982ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X....X...XX....X......XXXX..XX...X.....X.X....X..........] duration: 22.319906665s FAIL (2025-07-27 00:46:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.319941218s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X....] duration: 24.585366757s FAIL (2025-07-27 00:46:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 24.585398251s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X.X..X.X...............................X..................] duration: 2.85401791s FAIL (2025-07-27 00:46:15 UTC) orbiter warning: context deadline exceeded Duration: 2.85404321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX.........] duration: 2.980891987s FAIL (2025-07-27 00:46:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.980990401s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...X....X...XX....X......XXXX..XX...X.....X.X....X.........] duration: 3.218971744s FAIL (2025-07-27 00:46:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.219079279s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X...] duration: 2.357814824s FAIL (2025-07-27 00:45:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.357849275s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X.X..X.X...............................X.................] duration: 3.018285319s FAIL (2025-07-27 00:45:53 UTC) orbiter warning: context deadline exceeded Duration: 3.018309365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX.XX..XXXX.X...........X.XXXX.XX......XXXXX...XX.X...X.....] duration: 2.372373905s FAIL (2025-07-27 00:45:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== :41:07.222615952 (61h5m38.412914523s) [upload=58559]: Got next segment. Inline: false Jul 27 00:41:07.222678612 (61h5m38.41297719s) [upload=58559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:07.222684889 (61h5m38.412983462s) [upload=58559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:41:07.330155865 (61h5m38.520454446s) [seg_pos={0 0}, upload=58559]: Begin upload segment... Jul 27 00:41:07.330223641 (61h5m38.520522225s) [seg_pos={0 0}, upload=58559]: Done begin upload segment. Jul 27 00:41:07.330253409 (61h5m38.520551992s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.331859693 (61h5m38.522158283s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332330366 (61h5m38.52262895s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332739898 (61h5m38.523038466s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333152185 (61h5m38.523450752s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333682612 (61h5m38.523981181s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334071276 (61h5m38.524369844s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334428527 (61h5m38.524727095s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334781140 (61h5m38.52507972s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335157956 (61h5m38.525456544s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335505072 (61h5m38.525803652s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335862982 (61h5m38.526161564s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336223096 (61h5m38.526521679s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336566256 (61h5m38.526864838s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336924464 (61h5m38.527223047s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.337308564 (61h5m38.527607147s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.959715000 (61h5m39.150013589s) [upload=58559]: Next returned nil segment Jul 27 00:41:07.959737017 (61h5m39.150035606s) [upload=58559]: Waiting for error group managing segments... Jul 27 00:41:08.098609518 (61h5m39.288908087s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.114056645 (61h5m39.30435523s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115224636 (61h5m39.305523208s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115378353 (61h5m39.305676945s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.119763396 (61h5m39.310061969s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.131072499 (61h5m39.321371079s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.192043578 (61h5m39.382342152s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:09.562998275 (61h5m40.753296862s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:26.072685752 (61h5m57.262984318s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:36.947344132 (61h6m8.137642702s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947443385 (61h6m8.137741965s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection Jul 27 00:41:36.947487056 (61h6m8.137785647s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947752065 (61h6m8.138050651s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection Jul 27 00:41:36.947790491 (61h6m8.138089061s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection Jul 27 00:41:36.947845062 (61h6m8.138143632s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 139.59.144.22:49972->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:36.947886799 (61h6m8.138185382s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection Jul 27 00:41:36.947923797 (61h6m8.138222386s) [seg_pos={0 0}, upload=58559]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:41:37.055592756 (61h6m8.24589133s) [upload=58559]: Done uploading segments... Jul 27 00:41:37.055626444 (61h6m8.245925032s) [upload=58559]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.372428825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 908.001597ms FAIL (2025-07-27 00:45:42 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 908.038255ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX........] duration: 21.276270623s FAIL (2025-07-27 00:45:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.276300832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [.XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X..] duration: 4.601248164s FAIL (2025-07-27 00:45:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.60129042s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX...X....X...XX....X......XXXX..XX...X.....X.X....X........] duration: 3.137059877s FAIL (2025-07-27 00:45:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.137105761s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 958.437423ms FAIL (2025-07-27 00:45:02 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 958.558177ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X.X..X.X...............................X................] duration: 11.474593729s FAIL (2025-07-27 00:45:00 UTC) orbiter warning: context deadline exceeded Duration: 11.474655502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX...X....X...XX....X......XXXX..XX...X.....X.X....X.......] duration: 2.521150842s FAIL (2025-07-27 00:44:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.521689324s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX.......] duration: 3.828054359s FAIL (2025-07-27 00:44:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.828170708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX.XX..XXXX.X...........X.XXXX.XX......XXXXX...XX.X...X....] duration: 2.290653634s FAIL (2025-07-27 00:44:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== :41:07.222615952 (61h5m38.412914523s) [upload=58559]: Got next segment. Inline: false Jul 27 00:41:07.222678612 (61h5m38.41297719s) [upload=58559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:07.222684889 (61h5m38.412983462s) [upload=58559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:41:07.330155865 (61h5m38.520454446s) [seg_pos={0 0}, upload=58559]: Begin upload segment... Jul 27 00:41:07.330223641 (61h5m38.520522225s) [seg_pos={0 0}, upload=58559]: Done begin upload segment. Jul 27 00:41:07.330253409 (61h5m38.520551992s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.331859693 (61h5m38.522158283s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332330366 (61h5m38.52262895s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332739898 (61h5m38.523038466s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333152185 (61h5m38.523450752s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333682612 (61h5m38.523981181s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334071276 (61h5m38.524369844s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334428527 (61h5m38.524727095s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334781140 (61h5m38.52507972s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335157956 (61h5m38.525456544s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335505072 (61h5m38.525803652s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335862982 (61h5m38.526161564s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336223096 (61h5m38.526521679s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336566256 (61h5m38.526864838s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336924464 (61h5m38.527223047s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.337308564 (61h5m38.527607147s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.959715000 (61h5m39.150013589s) [upload=58559]: Next returned nil segment Jul 27 00:41:07.959737017 (61h5m39.150035606s) [upload=58559]: Waiting for error group managing segments... Jul 27 00:41:08.098609518 (61h5m39.288908087s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.114056645 (61h5m39.30435523s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115224636 (61h5m39.305523208s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115378353 (61h5m39.305676945s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.119763396 (61h5m39.310061969s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.131072499 (61h5m39.321371079s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.192043578 (61h5m39.382342152s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:09.562998275 (61h5m40.753296862s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:26.072685752 (61h5m57.262984318s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:36.947344132 (61h6m8.137642702s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947443385 (61h6m8.137741965s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection Jul 27 00:41:36.947487056 (61h6m8.137785647s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947752065 (61h6m8.138050651s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection Jul 27 00:41:36.947790491 (61h6m8.138089061s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection Jul 27 00:41:36.947845062 (61h6m8.138143632s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 139.59.144.22:49972->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:36.947886799 (61h6m8.138185382s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection Jul 27 00:41:36.947923797 (61h6m8.138222386s) [seg_pos={0 0}, upload=58559]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:41:37.055592756 (61h6m8.24589133s) [upload=58559]: Done uploading segments... Jul 27 00:41:37.055626444 (61h6m8.245925032s) [upload=58559]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.290693835s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X..X.X...............................X...............] duration: 3.832756414s FAIL (2025-07-27 00:44:37 UTC) orbiter warning: context deadline exceeded Duration: 3.832780785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX......] duration: 3.067498521s FAIL (2025-07-27 00:44:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.067618267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 941.649194ms FAIL (2025-07-27 00:44:12 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 941.680951ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X.] duration: 2.575333754s FAIL (2025-07-27 00:44:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.575365692s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX...X....X...XX....X......XXXX..XX...X.....X.X....X......] duration: 2.62846501s FAIL (2025-07-27 00:44:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.628489938s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X.X..X.X...............................X..............] duration: 4.108347136s FAIL (2025-07-27 00:44:08 UTC) orbiter warning: context deadline exceeded Duration: 4.108379621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [X..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.X] duration: 30.316324615s FAIL (2025-07-27 00:44:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.316356483s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X.X..X.X...............................X.............] duration: 43.667146686s FAIL (2025-07-27 00:44:03 UTC) orbiter warning: context deadline exceeded Duration: 43.667190477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XX...X....X...XX....X......XXXX..XX...X.....X.X....X.....] duration: 2.874265698s FAIL (2025-07-27 00:43:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.874296532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [.X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX.....] duration: 27.632614258s FAIL (2025-07-27 00:43:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 27.632647324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XX.XX..XXXX.X...........X.XXXX.XX......XXXXX...XX.X...X...] duration: 11.796814079s FAIL (2025-07-27 00:43:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== :41:07.222615952 (61h5m38.412914523s) [upload=58559]: Got next segment. Inline: false Jul 27 00:41:07.222678612 (61h5m38.41297719s) [upload=58559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:07.222684889 (61h5m38.412983462s) [upload=58559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:41:07.330155865 (61h5m38.520454446s) [seg_pos={0 0}, upload=58559]: Begin upload segment... Jul 27 00:41:07.330223641 (61h5m38.520522225s) [seg_pos={0 0}, upload=58559]: Done begin upload segment. Jul 27 00:41:07.330253409 (61h5m38.520551992s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.331859693 (61h5m38.522158283s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332330366 (61h5m38.52262895s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332739898 (61h5m38.523038466s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333152185 (61h5m38.523450752s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333682612 (61h5m38.523981181s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334071276 (61h5m38.524369844s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334428527 (61h5m38.524727095s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334781140 (61h5m38.52507972s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335157956 (61h5m38.525456544s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335505072 (61h5m38.525803652s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335862982 (61h5m38.526161564s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336223096 (61h5m38.526521679s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336566256 (61h5m38.526864838s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336924464 (61h5m38.527223047s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.337308564 (61h5m38.527607147s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.959715000 (61h5m39.150013589s) [upload=58559]: Next returned nil segment Jul 27 00:41:07.959737017 (61h5m39.150035606s) [upload=58559]: Waiting for error group managing segments... Jul 27 00:41:08.098609518 (61h5m39.288908087s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.114056645 (61h5m39.30435523s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115224636 (61h5m39.305523208s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115378353 (61h5m39.305676945s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.119763396 (61h5m39.310061969s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.131072499 (61h5m39.321371079s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.192043578 (61h5m39.382342152s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:09.562998275 (61h5m40.753296862s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:26.072685752 (61h5m57.262984318s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:36.947344132 (61h6m8.137642702s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947443385 (61h6m8.137741965s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection Jul 27 00:41:36.947487056 (61h6m8.137785647s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947752065 (61h6m8.138050651s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection Jul 27 00:41:36.947790491 (61h6m8.138089061s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection Jul 27 00:41:36.947845062 (61h6m8.138143632s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 139.59.144.22:49972->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:36.947886799 (61h6m8.138185382s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection Jul 27 00:41:36.947923797 (61h6m8.138222386s) [seg_pos={0 0}, upload=58559]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:41:37.055592756 (61h6m8.24589133s) [upload=58559]: Done uploading segments... Jul 27 00:41:37.055626444 (61h6m8.245925032s) [upload=58559]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 11.796877967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 860.11548ms FAIL (2025-07-27 00:43:12 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 860.22696ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 860.631275ms FAIL (2025-07-27 00:42:57 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 860.676607ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..XX.XX..XXXX.X...........X.XXXX.XX......XXXXX...XX.X...X..] duration: 2.392714795s FAIL (2025-07-27 00:42:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== :41:07.222615952 (61h5m38.412914523s) [upload=58559]: Got next segment. Inline: false Jul 27 00:41:07.222678612 (61h5m38.41297719s) [upload=58559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:07.222684889 (61h5m38.412983462s) [upload=58559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:41:07.330155865 (61h5m38.520454446s) [seg_pos={0 0}, upload=58559]: Begin upload segment... Jul 27 00:41:07.330223641 (61h5m38.520522225s) [seg_pos={0 0}, upload=58559]: Done begin upload segment. Jul 27 00:41:07.330253409 (61h5m38.520551992s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.331859693 (61h5m38.522158283s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332330366 (61h5m38.52262895s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332739898 (61h5m38.523038466s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333152185 (61h5m38.523450752s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333682612 (61h5m38.523981181s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334071276 (61h5m38.524369844s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334428527 (61h5m38.524727095s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334781140 (61h5m38.52507972s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335157956 (61h5m38.525456544s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335505072 (61h5m38.525803652s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335862982 (61h5m38.526161564s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336223096 (61h5m38.526521679s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336566256 (61h5m38.526864838s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336924464 (61h5m38.527223047s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.337308564 (61h5m38.527607147s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.959715000 (61h5m39.150013589s) [upload=58559]: Next returned nil segment Jul 27 00:41:07.959737017 (61h5m39.150035606s) [upload=58559]: Waiting for error group managing segments... Jul 27 00:41:08.098609518 (61h5m39.288908087s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.114056645 (61h5m39.30435523s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115224636 (61h5m39.305523208s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115378353 (61h5m39.305676945s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.119763396 (61h5m39.310061969s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.131072499 (61h5m39.321371079s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.192043578 (61h5m39.382342152s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:09.562998275 (61h5m40.753296862s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:26.072685752 (61h5m57.262984318s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:36.947344132 (61h6m8.137642702s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947443385 (61h6m8.137741965s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection Jul 27 00:41:36.947487056 (61h6m8.137785647s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947752065 (61h6m8.138050651s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection Jul 27 00:41:36.947790491 (61h6m8.138089061s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection Jul 27 00:41:36.947845062 (61h6m8.138143632s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 139.59.144.22:49972->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:36.947886799 (61h6m8.138185382s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection Jul 27 00:41:36.947923797 (61h6m8.138222386s) [seg_pos={0 0}, upload=58559]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:41:37.055592756 (61h6m8.24589133s) [upload=58559]: Done uploading segments... Jul 27 00:41:37.055626444 (61h6m8.245925032s) [upload=58559]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.392776817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [XX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX.] duration: 2.396452198s FAIL (2025-07-27 00:42:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.396485748s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X.X..X.X...............................X............] duration: 6.518849737s FAIL (2025-07-27 00:42:47 UTC) orbiter warning: context deadline exceeded Duration: 6.518868185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX...X....X...XX....X......XXXX..XX...X.....X.X....X....] duration: 3.237350951s FAIL (2025-07-27 00:42:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.237377384s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........X.X..X.X...............................X...........] duration: 40.75013358s FAIL (2025-07-27 00:42:40 UTC) orbiter warning: context deadline exceeded Duration: 40.750164449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [..X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX....] duration: 22.332984569s FAIL (2025-07-27 00:42:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.333013807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.XX] duration: 30.116436017s FAIL (2025-07-27 00:42:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.116483305s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.X.X.XX...] duration: 53.75089216s FAIL (2025-07-27 00:42:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 88913508s) [upload=58519]: Starting upload Jul 27 00:38:26.198707515 (61h2m57.389006105s) [upload=58519]: Done starting upload Jul 27 00:38:26.199117666 (61h2m57.389416246s) [upload=58519]: Uploading segments... Jul 27 00:38:26.199186301 (61h2m57.389484871s) [upload=58519]: Got next segment. Inline: false Jul 27 00:38:26.199299628 (61h2m57.389598208s) [upload=58519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:38:26.199305355 (61h2m57.389603928s) [upload=58519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:38:26.394877781 (61h2m57.585176379s) [seg_pos={0 0}, upload=58519]: Begin upload segment... Jul 27 00:38:26.394944689 (61h2m57.585243261s) [seg_pos={0 0}, upload=58519]: Done begin upload segment. Jul 27 00:38:26.394974627 (61h2m57.585273216s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.394985864 (61h2m57.585284448s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395054240 (61h2m57.585352806s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395522408 (61h2m57.585820998s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395531198 (61h2m57.585829787s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395628062 (61h2m57.58592665s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.395990917 (61h2m57.586289507s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396000917 (61h2m57.586299506s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396362824 (61h2m57.586661407s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.396816336 (61h2m57.587114924s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397390475 (61h2m57.587689065s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.397777892 (61h2m57.588076479s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398194082 (61h2m57.588492672s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398582082 (61h2m57.588880664s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.398962615 (61h2m57.589261195s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399026562 (61h2m57.589325152s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399483285 (61h2m57.589781878s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.399984258 (61h2m57.590282847s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Uploading piece... Jul 27 00:38:26.778716400 (61h2m57.969014989s) [noise=true , address=77.90.31.13:4656 , piece_id=IABUSZ33BPXY2ZMYF22OAOWJVZEZY7HQACYONVE4M7MCDIXS7MIQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:26.958413398 (61h2m58.148711964s) [upload=58519]: Next returned nil segment Jul 27 00:38:26.958431699 (61h2m58.148730264s) [upload=58519]: Waiting for error group managing segments... Jul 27 00:38:27.134142936 (61h2m58.324441509s) [noise=true , address=77.90.31.16:4747 , piece_id=GRZJP2VQI7D26W6LCS5FVLWZZXEEM73NPIXDACGMQOKFKYBODQOQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.427187251 (61h2m58.617485837s) [noise=true , address=64.49.13.140:4508 , piece_id=HIABJPVNTWGRBS5VOYIIORAV4LGIBTGZDRXLTPH62FZG5BNC5ZZA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:27.471434377 (61h2m58.661733018s) [noise=true , address=79.127.219.38:4006 , piece_id=VBI2ZUSH2SJZD5AC3TJILDOOWWZJRUKARCIK3IXOSZ5D2Z7EWKUA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.653419745 (61h2m58.84371832s) [noise=true , address=79.127.205.246:4285 , piece_id=YRZVGNDX746XGEXNELOR5NG3LRUV3XS5TC7ALFXIFEHCF5UQAEVQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.664342604 (61h2m58.854641233s) [noise=true , address=79.127.205.235:4676 , piece_id=I57GDTSKRX44VNPOOZI4UGUST7HSK6MQ7ETHUPPWG3F6LZ43KF4A, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:27.710124479 (61h2m58.900423059s) [noise=true , address=79.127.205.245:4247 , piece_id=KM46JZ7RSINUP3ML5KABHPECCMIRJD353EDSDWVUZMYJ4GK5FEDQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:28.871765427 (61h3m0.06206403s) [noise=true , address=45.91.116.12:4231 , piece_id=SEYFIHN35SBNU5BQOVFWHIMG3OFHPKUOQA46D3Y47V5DTIJRZTYA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: storage node overloaded, request limit: 2000 Jul 27 00:38:29.918062454 (61h3m1.108361026s) [noise=true , address=156.146.43.225:4448 , piece_id=HFLTO2IBC67KFMPZSKNU6MKOU225FAELEK2CLQBYFIFRN2KJDRCQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:33.113873628 (61h3m4.30417221s) [noise=true , address=45.91.116.14:4194 , piece_id=QPBCQMBTUC5H4CDREIC4I6MXP5NJB6OTJ34IN7PUR5YCNMJBRVAA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:36.137900605 (61h3m7.328199191s) [noise=true , address=79.127.220.101:4041 , piece_id=AXUQMTSC77JNMPC22IWRLXTXV4OXZ3GIYUCGCOV7AY5C4EJOM5AA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:38:39.153534369 (61h3m10.343832944s) [noise=true , address=104.36.82.8:4979 , piece_id=YEH2J3ESUAHYSXX4EGWSJPYHOHNTXKZUYDJVB2E6QGFZUR3FKWXA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:04.842619937 (61h3m36.032918508s) [noise=true , address=77.90.31.12:4605 , piece_id=42LHUC6IZJ7NXGTV5MXY47AYQJX676TIUUFIHYMSZKILOP5KVZBA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:10.789076833 (61h3m41.979375416s) [noise=true , address=77.90.31.17:4780 , piece_id=C2FYY5NOH4QXRBTWWODAIT2YX5SGBVPKHCZLLTNQE7F4R2KUHABA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:15.930448411 (61h3m47.120746978s) [noise=true , address=104.36.82.9:4028 , piece_id=E52D6BMCWC2SISMLLQHRMWCJSTRVKNSZWK7AGTXL7FHWWXTSHQEA, seg_pos={0 0}, upload=58519]: Done uploading piece. err: <nil> Jul 27 00:39:25.928450241 (61h3m57.118748821s) [noise=true , address=104.36.82.46:4507 , piece_id=QBG3E2SB5AR7F2YAPGAOPS62AZUFMKQNY56VZ7XXPFVMW2O2T4DQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928480032 (61h3m57.118778603s) [noise=true , address=104.36.82.54:4162 , piece_id=UJREDUDUKZMLVJEV5THCV2TUK73YHQH5YSVIVB4RLYV573Z2QENQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:39:25.928622077 (61h3m57.118920664s) [noise=true , address=104.36.82.47:4524 , piece_id=5TPVENO44QOJOQF5SSFOE556ME2I5M4RKZYPAVRXIOPIQVIU7XLQ, seg_pos={0 0}, upload=58519]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection; write tcp 139.59.144.22:59922->104.36.82.47:4524: use of closed network connection Jul 27 00:39:25.928664572 (61h3m57.118963161s) [seg_pos={0 0}, upload=58519]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:39:26.037001335 (61h3m57.227299903s) [upload=58519]: Done uploading segments... Jul 27 00:39:26.037027944 (61h3m57.227326524s) [upload=58519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 53.750918111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....XX...X....X...XX....X......XXXX..XX...X.....X.X....X...] duration: 2.623241322s FAIL (2025-07-27 00:41:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.623285661s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 835.965624ms FAIL (2025-07-27 00:41:50 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 836.017507ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.XXXX.X..X.X] duration: 30.003089559s FAIL (2025-07-27 00:41:47 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.003119698s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX..XX.XX..XXXX.X...........X.XXXX.XX......XXXXX...XX.X...X.] duration: 2.339778446s FAIL (2025-07-27 00:41:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== :41:07.222615952 (61h5m38.412914523s) [upload=58559]: Got next segment. Inline: false Jul 27 00:41:07.222678612 (61h5m38.41297719s) [upload=58559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:07.222684889 (61h5m38.412983462s) [upload=58559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:41:07.330155865 (61h5m38.520454446s) [seg_pos={0 0}, upload=58559]: Begin upload segment... Jul 27 00:41:07.330223641 (61h5m38.520522225s) [seg_pos={0 0}, upload=58559]: Done begin upload segment. Jul 27 00:41:07.330253409 (61h5m38.520551992s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.331859693 (61h5m38.522158283s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332330366 (61h5m38.52262895s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332739898 (61h5m38.523038466s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333152185 (61h5m38.523450752s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333682612 (61h5m38.523981181s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334071276 (61h5m38.524369844s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334428527 (61h5m38.524727095s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334781140 (61h5m38.52507972s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335157956 (61h5m38.525456544s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335505072 (61h5m38.525803652s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335862982 (61h5m38.526161564s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336223096 (61h5m38.526521679s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336566256 (61h5m38.526864838s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336924464 (61h5m38.527223047s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.337308564 (61h5m38.527607147s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.959715000 (61h5m39.150013589s) [upload=58559]: Next returned nil segment Jul 27 00:41:07.959737017 (61h5m39.150035606s) [upload=58559]: Waiting for error group managing segments... Jul 27 00:41:08.098609518 (61h5m39.288908087s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.114056645 (61h5m39.30435523s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115224636 (61h5m39.305523208s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115378353 (61h5m39.305676945s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.119763396 (61h5m39.310061969s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.131072499 (61h5m39.321371079s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.192043578 (61h5m39.382342152s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:09.562998275 (61h5m40.753296862s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:26.072685752 (61h5m57.262984318s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:36.947344132 (61h6m8.137642702s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947443385 (61h6m8.137741965s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection Jul 27 00:41:36.947487056 (61h6m8.137785647s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947752065 (61h6m8.138050651s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection Jul 27 00:41:36.947790491 (61h6m8.138089061s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection Jul 27 00:41:36.947845062 (61h6m8.138143632s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 139.59.144.22:49972->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:36.947886799 (61h6m8.138185382s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection Jul 27 00:41:36.947923797 (61h6m8.138222386s) [seg_pos={0 0}, upload=58559]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:41:37.055592756 (61h6m8.24589133s) [upload=58559]: Done uploading segments... Jul 27 00:41:37.055626444 (61h6m8.245925032s) [upload=58559]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.339814892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXX..XX.XX..XXXX.X...........X.XXXX.XX......XXXXX...XX.X...X] duration: 30.110299485s FAIL (2025-07-27 00:41:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== :41:07.222615952 (61h5m38.412914523s) [upload=58559]: Got next segment. Inline: false Jul 27 00:41:07.222678612 (61h5m38.41297719s) [upload=58559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:41:07.222684889 (61h5m38.412983462s) [upload=58559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:41:07.330155865 (61h5m38.520454446s) [seg_pos={0 0}, upload=58559]: Begin upload segment... Jul 27 00:41:07.330223641 (61h5m38.520522225s) [seg_pos={0 0}, upload=58559]: Done begin upload segment. Jul 27 00:41:07.330253409 (61h5m38.520551992s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.331859693 (61h5m38.522158283s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332330366 (61h5m38.52262895s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.332739898 (61h5m38.523038466s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333152185 (61h5m38.523450752s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.333682612 (61h5m38.523981181s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334071276 (61h5m38.524369844s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334428527 (61h5m38.524727095s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.334781140 (61h5m38.52507972s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335157956 (61h5m38.525456544s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335505072 (61h5m38.525803652s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.335862982 (61h5m38.526161564s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336223096 (61h5m38.526521679s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336566256 (61h5m38.526864838s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.336924464 (61h5m38.527223047s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.337308564 (61h5m38.527607147s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Uploading piece... Jul 27 00:41:07.959715000 (61h5m39.150013589s) [upload=58559]: Next returned nil segment Jul 27 00:41:07.959737017 (61h5m39.150035606s) [upload=58559]: Waiting for error group managing segments... Jul 27 00:41:08.098609518 (61h5m39.288908087s) [noise=true , address=45.91.116.4:4940 , piece_id=PQBNNCFR3WC7PFIM6SZGAVYX6PFP6U2OJSVIYHSQQUTUKHNBE4XA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.114056645 (61h5m39.30435523s) [noise=true , address=45.91.116.13:4153 , piece_id=MIBKQISRDZKRJSDUGXBU2Y52VAGICCMAJDGYNL6M2XQARNA25GAQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115224636 (61h5m39.305523208s) [noise=true , address=77.90.31.16:4735 , piece_id=46JFJ7WIGGEEPMGIPGQ3YYRS36J6BQOXYRT3ZPJW27KUNJKJH4GA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.115378353 (61h5m39.305676945s) [noise=true , address=77.90.31.16:4733 , piece_id=LIW7PFNEUHDSQCQTTAHBV6NEPOXCXU6ZSRI5JS6HNRWK4X6VUMEA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.119763396 (61h5m39.310061969s) [noise=true , address=45.91.116.6:4973 , piece_id=THXYG3NNU6CREBL7ZZ2PDSV5KFGA4XW3X2RYSFHBFIZRI2YHBPNQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.131072499 (61h5m39.321371079s) [noise=true , address=45.91.116.14:4190 , piece_id=IEM7SFUTY6YMJD6N6EDK3U6JQDCGIMFWMCOEFJ5CB3OW4FRFHR7A, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:08.192043578 (61h5m39.382342152s) [noise=true , address=45.91.116.5:4299 , piece_id=P6QR6HS7VUUUEZV3B7BJMXI4HXDMLY3BE3BEMLXTZKH6PAXBS7OA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:09.562998275 (61h5m40.753296862s) [noise=true , address=77.90.31.10:4518 , piece_id=EMDJMZT6UWN7ANHE3S63NUTFLQBP2XPWMVDRKCJDWDZQJXIANASA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:26.072685752 (61h5m57.262984318s) [noise=true , address=104.36.82.4:4801 , piece_id=HOG3BG4I6WHFZOG3L6HOTLYWVQJNBZJBSFNT3WMASHP3QOTLRUGA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: <nil> Jul 27 00:41:36.947344132 (61h6m8.137642702s) [noise=true , address=104.36.82.14:4222 , piece_id=YJ7TYPVEHB5FXIOP4S3FKBX4ZAC4YEU5UGL2UGUHNGZUAPFTD6JA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947443385 (61h6m8.137741965s) [noise=true , address=104.36.82.10:4079 , piece_id=WKUVSMS5DTK3ZFCLLPDPPGNYUD5NYIMB35EUG65J773QUDA4CPYA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53038->104.36.82.10:4079: use of closed network connection Jul 27 00:41:36.947487056 (61h6m8.137785647s) [noise=true , address=104.36.82.6:4874 , piece_id=LZUZ2LD34IYRLYTU7YQOGQG3C6H2NWOEZPUUSS7Z5XSMQRRHJT7Q, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:41:36.947752065 (61h6m8.138050651s) [noise=true , address=104.36.82.5:4839 , piece_id=HIDRHA2GWD4G7S3A5MFMXWXUEMMYB3JZ26OWYOZQN6PP5JOTANPQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38238->104.36.82.5:4839: use of closed network connection Jul 27 00:41:36.947790491 (61h6m8.138089061s) [noise=true , address=104.36.82.12:4151 , piece_id=2MYG27EANQMNIOJOHBRRVL2K5NYXXDLH4WMJV4CTNE7MCIO2NGWQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44774->104.36.82.12:4151: use of closed network connection Jul 27 00:41:36.947845062 (61h6m8.138143632s) [noise=true , address=77.90.31.17:4771 , piece_id=2L5N6ONJJWPJBHW5TMYSVDVJP2EDKDJKP4ZROJ562EVHIHAD7EVA, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 139.59.144.22:49972->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:41:36.947886799 (61h6m8.138185382s) [noise=true , address=104.36.82.2:4730 , piece_id=UVVGMRJQB43DBBBANFZ6PHINRK4VHCRC7ZYZRVAH6WVIGHXLZLFQ, seg_pos={0 0}, upload=58559]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44434->104.36.82.2:4730: use of closed network connection Jul 27 00:41:36.947923797 (61h6m8.138222386s) [seg_pos={0 0}, upload=58559]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:41:37.055592756 (61h6m8.24589133s) [upload=58559]: Done uploading segments... Jul 27 00:41:37.055626444 (61h6m8.245925032s) [upload=58559]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.110380562s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)