Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...X.X.X..X..XX] duration: 1m0.106284506s FAIL (2025-07-26 23:59:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 041423106 (60h22m50.231721691s) [upload=57883]: Done starting upload Jul 26 23:58:19.041549142 (60h22m50.231847709s) [upload=57883]: Uploading segments... Jul 26 23:58:19.041818163 (60h22m50.23211675s) [upload=57883]: Got next segment. Inline: false Jul 26 23:58:19.041875047 (60h22m50.232173633s) [upload=57883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:58:19.041892881 (60h22m50.232191471s) [upload=57883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:58:19.239310850 (60h22m50.429609441s) [seg_pos={0 0}, upload=57883]: Begin upload segment... Jul 26 23:58:19.239412609 (60h22m50.429711197s) [noise=true , address=104.36.82.56:4249 , piece_id=K2YORWNIRDTZOWN2EJICZ6BA2H2DIGE5YHYOG7JLXBXTLMN2RERA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.239829621 (60h22m50.430128228s) [noise=true , address=156.146.43.225:4434 , piece_id=J4PON6UWOYYTZ5H2FROE6WT2JT5RQF6GIE73SXIW2VB5BZLDBXIA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.242070035 (60h22m50.432368621s) [seg_pos={0 0}, upload=57883]: Done begin upload segment. Jul 26 23:58:19.242149917 (60h22m50.432448506s) [noise=true , address=79.127.220.101:4048 , piece_id=WRB7TW4W7LEPRI5Q7OCNGT3BVKRLDYYGV2FTRMOMHFEF6HCID5VA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.242646298 (60h22m50.432944889s) [noise=true , address=79.127.201.215:4352 , piece_id=P7ZNK64ULPEIWEOCMAWS7B3SRJ5PHMJBFN5TY3WGB3HYOZ2E2WRQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.243031713 (60h22m50.433330307s) [noise=true , address=64.49.13.26:4953 , piece_id=BMYFKLIKYSMN2K44ZQU5FZ6ANGSJPKITVX74T7EU5S3KCBC2PBFQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.243422029 (60h22m50.433720617s) [noise=true , address=45.91.116.14:4206 , piece_id=VS5W4KNY36LQRM4BP77OIK7JJJWEM7S454YBR4D24TX4P2WEYHRA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.243813140 (60h22m50.43411173s) [noise=true , address=79.127.219.38:4005 , piece_id=CZH2CXFGLQDFRQUVWMOWMVSF74NTKVSGVUX4QFMTXEG2DVKBFGEA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.244183544 (60h22m50.43448213s) [noise=true , address=79.127.205.245:4246 , piece_id=HUPEM4XZ7CZTK2ZKLKYCXQ54TGYUYHL27W2FG7EO5NRCVMCLL5SQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.244567491 (60h22m50.43486608s) [noise=true , address=77.90.31.18:4821 , piece_id=W6H6JV5HVMS2FDDVEASMBM7CGFMA3PE3DNDVLHPSUXJ5XWZL3JQQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.245109227 (60h22m50.435407821s) [noise=true , address=64.49.13.140:4514 , piece_id=A4VIMSXTJIZC4HCHEHCIXWJSCINYL46NY535IOOOWNNH3ATNXFPA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.245541789 (60h22m50.435840388s) [noise=true , address=79.127.205.235:4688 , piece_id=3JLCPYNXCPVP7PQY4QUKM7T2O4DAGIMEJV7Y3L7DXGHKKURGZJRA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.245949176 (60h22m50.436247758s) [noise=true , address=77.90.31.15:4705 , piece_id=7V5G3RPNEXRNIXFQ2B57YQ3AXI3JJUKLJLTODVSJYJIUXFETGRWQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.246323756 (60h22m50.436622345s) [noise=true , address=64.49.13.138:4482 , piece_id=KHTF2STE5LNTD4CR33MYLPUFIN4QJQ7YMRN7PRI7ZLRPHOGVQZTA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.246660269 (60h22m50.436958851s) [noise=true , address=104.36.82.52:4106 , piece_id=7NE7H4O3IOBT6ZX5NV7ETH6WOM3QZQRLTNAQRMVR523DQGBOCAGQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.246991102 (60h22m50.437289691s) [noise=true , address=104.36.82.46:4483 , piece_id=6V76FLHAOFDQ2YR7L7RNLRBUGC4QEK7JNXIID2HKOJCL2TW6QYOQ, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.247362465 (60h22m50.437661054s) [noise=true , address=79.127.201.215:4353 , piece_id=ULTRPFA5IBN6H2FWX6QBATOTXCUDG25WSCOFPHR7FRVPGYE2XQPA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.247781172 (60h22m50.438079753s) [noise=true , address=104.36.82.47:4528 , piece_id=GO2PEI52VQF6JZ4KJMKUFMP3KPFGRYKSBX2F4RJGFKUJ7FHVZK7A, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.248127425 (60h22m50.438426012s) [noise=true , address=79.127.205.233:4641 , piece_id=NSSRJDGTTGYXZZI2E7D7KXK4PGHOKLICRQOZXEFPWXPM5CAI7CKA, seg_pos={0 0}, upload=57883]: Uploading piece... Jul 26 23:58:19.986871614 (60h22m51.177170205s) [upload=57883]: Next returned nil segment Jul 26 23:58:19.986893609 (60h22m51.177192198s) [upload=57883]: Waiting for error group managing segments... Jul 26 23:58:20.094834187 (60h22m51.285132769s) [noise=true , address=64.49.13.138:4482 , piece_id=KHTF2STE5LNTD4CR33MYLPUFIN4QJQ7YMRN7PRI7ZLRPHOGVQZTA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:58:20.289500049 (60h22m51.479798629s) [noise=true , address=79.127.219.38:4005 , piece_id=CZH2CXFGLQDFRQUVWMOWMVSF74NTKVSGVUX4QFMTXEG2DVKBFGEA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:20.538818794 (60h22m51.729117381s) [noise=true , address=64.49.13.26:4953 , piece_id=BMYFKLIKYSMN2K44ZQU5FZ6ANGSJPKITVX74T7EU5S3KCBC2PBFQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:20.551894853 (60h22m51.742193437s) [noise=true , address=79.127.205.233:4641 , piece_id=NSSRJDGTTGYXZZI2E7D7KXK4PGHOKLICRQOZXEFPWXPM5CAI7CKA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:20.608761824 (60h22m51.799060409s) [noise=true , address=79.127.205.235:4688 , piece_id=3JLCPYNXCPVP7PQY4QUKM7T2O4DAGIMEJV7Y3L7DXGHKKURGZJRA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:20.610504867 (60h22m51.800803435s) [noise=true , address=79.127.205.245:4246 , piece_id=HUPEM4XZ7CZTK2ZKLKYCXQ54TGYUYHL27W2FG7EO5NRCVMCLL5SQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:21.139933705 (60h22m52.330232272s) [noise=true , address=79.127.220.101:4048 , piece_id=WRB7TW4W7LEPRI5Q7OCNGT3BVKRLDYYGV2FTRMOMHFEF6HCID5VA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:22.662371414 (60h22m53.852669983s) [noise=true , address=156.146.43.225:4434 , piece_id=J4PON6UWOYYTZ5H2FROE6WT2JT5RQF6GIE73SXIW2VB5BZLDBXIA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:22.896596858 (60h22m54.086895428s) [noise=true , address=64.49.13.140:4514 , piece_id=A4VIMSXTJIZC4HCHEHCIXWJSCINYL46NY535IOOOWNNH3ATNXFPA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:40.691622523 (60h23m11.881921094s) [noise=true , address=104.36.82.56:4249 , piece_id=K2YORWNIRDTZOWN2EJICZ6BA2H2DIGE5YHYOG7JLXBXTLMN2RERA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:46.155722579 (60h23m17.34602115s) [noise=true , address=104.36.82.46:4483 , piece_id=6V76FLHAOFDQ2YR7L7RNLRBUGC4QEK7JNXIID2HKOJCL2TW6QYOQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:58:59.050059155 (60h23m30.240357728s) [noise=true , address=104.36.82.52:4106 , piece_id=7NE7H4O3IOBT6ZX5NV7ETH6WOM3QZQRLTNAQRMVR523DQGBOCAGQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:59:12.264519110 (60h23m43.454817691s) [noise=true , address=79.127.201.215:4353 , piece_id=ULTRPFA5IBN6H2FWX6QBATOTXCUDG25WSCOFPHR7FRVPGYE2XQPA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: <nil> Jul 26 23:59:18.770282142 (60h23m49.960580732s) [noise=true , address=77.90.31.15:4705 , piece_id=7V5G3RPNEXRNIXFQ2B57YQ3AXI3JJUKLJLTODVSJYJIUXFETGRWQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw, address:77.90.31.15:4705): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:59:18.770357645 (60h23m49.960656224s) [noise=true , address=79.127.201.215:4352 , piece_id=P7ZNK64ULPEIWEOCMAWS7B3SRJ5PHMJBFN5TY3WGB3HYOZ2E2WRQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:12KNKWRTR6pZfxWFcFwsEKaqGsUMBBSTBsxvmeKsV1GDfEBvQk7, address:79.127.201.215:4352): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:59:18.770453667 (60h23m49.960752253s) [noise=true , address=45.91.116.14:4206 , piece_id=VS5W4KNY36LQRM4BP77OIK7JJJWEM7S454YBR4D24TX4P2WEYHRA, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:59:18.770523129 (60h23m49.960821695s) [noise=true , address=104.36.82.47:4528 , piece_id=GO2PEI52VQF6JZ4KJMKUFMP3KPFGRYKSBX2F4RJGFKUJ7FHVZK7A, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:59:18.770697920 (60h23m49.960996488s) [noise=true , address=77.90.31.18:4821 , piece_id=W6H6JV5HVMS2FDDVEASMBM7CGFMA3PE3DNDVLHPSUXJ5XWZL3JQQ, seg_pos={0 0}, upload=57883]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 139.59.144.22:40906->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:59:18.770734743 (60h23m49.961033332s) [seg_pos={0 0}, upload=57883]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:59:18.874523616 (60h23m50.064822184s) [upload=57883]: Done uploading segments... Jul 26 23:59:18.874562172 (60h23m50.064860751s) [upload=57883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.106305078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.XX.XXXXX] duration: 30.114715722s FAIL (2025-07-26 23:59:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ZSC6TXBQ, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.990781107 (60h23m12.181079693s) [noise=true , address=45.91.116.9:4263 , piece_id=HA43UJWPUMYJVV3DPS5IZTXCXOIFGR5JI3V6QIC6DOFX2AQ6YQFA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.991143939 (60h23m12.181442518s) [noise=true , address=77.90.31.18:4805 , piece_id=O26N35M2TLTJVWSHV5BBPA3QQK6LT3AXSD6CLZMH3UGYNSVELQKA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.991300464 (60h23m12.181599043s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMVL64RDMPBHPUBOULZA75U4OSZEWVFXGJ6LZBW4NCRQC3NNVKA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.991456977 (60h23m12.181755582s) [noise=true , address=77.90.31.14:4662 , piece_id=FP2HKN3AWPQ34I2FYW5B377ZWNPCBATJCQJ7NYLGELZEAU4AY6VA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.991702357 (60h23m12.182000944s) [noise=true , address=77.90.31.16:4735 , piece_id=CRUIQTI2X2KJUAMASHI2GCAEST4VSKBTGYWWLXURDDM2DY3DJ7TQ, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.991785936 (60h23m12.182084591s) [noise=true , address=45.91.116.12:4227 , piece_id=M5IKNOEODPJ5LLTCSP7ETVYPJIK25IEI3N4GITVD7ZVMMCBKMVMQ, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.992100407 (60h23m12.182398986s) [noise=true , address=45.91.116.11:4119 , piece_id=QTT3QS6UPUDYOEJOLLAHGAOE5VS2576RQX66O63EL6TTQ35T3DTA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.992134020 (60h23m12.182432599s) [noise=true , address=104.36.82.2:4732 , piece_id=EJCSRTNPIYJQBOCQXT2PFFOFO4AMSBR6G67CPTB5KGWI7BN5YUKQ, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.992429668 (60h23m12.182728253s) [noise=true , address=104.36.82.10:4079 , piece_id=TV6GL7EC7YCND76YCIKHB22QOQ7Z3AZKMFU2ZYKMARO2PMGRHSCA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.992513506 (60h23m12.182812092s) [noise=true , address=45.91.116.4:4940 , piece_id=JZ4PBGYFMTWV5ASCLPZJP26VIPZC7U2H3W5HL4CITAA2LC5HOGBQ, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.992776418 (60h23m12.183074997s) [noise=true , address=45.91.116.10:4081 , piece_id=OOSCSLRMBNR2HGMLH6JCQFGMSN2K6MVQEP32MY7ZXBRS723A63FQ, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.992810969 (60h23m12.183109548s) [noise=true , address=104.36.82.9:4005 , piece_id=HXZ2MEAADB4J3KYPQYY276NME4HYDAS3TWKAPL4RRKYU7Q4YYBIA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.993212890 (60h23m12.183511478s) [noise=true , address=77.90.31.17:4770 , piece_id=6VV45BQLIEMG7YMYMCN5YWRZDDZN3QBGQ3JDTF3GWSIDZVVVIM3Q, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.993218839 (60h23m12.183517411s) [noise=true , address=45.91.116.13:4153 , piece_id=YI5N7Y2OPSPVJ2DEERT3WZKEK6NOAGAJJLP7HDMKEIKWEGGEDATA, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:40.993298839 (60h23m12.183597423s) [noise=true , address=104.36.82.7:4909 , piece_id=CQQZHS3NJWZH625UBITYJNNIXHWRWSHYGKMJMFU5ZD7XCUVDC56Q, seg_pos={0 0}, upload=57889]: Uploading piece... Jul 26 23:58:47.337655468 (60h23m18.527954059s) [upload=57889]: Next returned nil segment Jul 26 23:58:47.337677223 (60h23m18.527975811s) [upload=57889]: Waiting for error group managing segments... Jul 26 23:58:48.447079683 (60h23m19.637378259s) [noise=true , address=104.36.82.10:4079 , piece_id=TV6GL7EC7YCND76YCIKHB22QOQ7Z3AZKMFU2ZYKMARO2PMGRHSCA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: <nil> Jul 26 23:58:54.975060005 (60h23m26.165358571s) [noise=true , address=104.36.82.7:4909 , piece_id=CQQZHS3NJWZH625UBITYJNNIXHWRWSHYGKMJMFU5ZD7XCUVDC56Q, seg_pos={0 0}, upload=57889]: Done uploading piece. err: <nil> Jul 26 23:59:01.659928882 (60h23m32.850227461s) [noise=true , address=45.91.116.12:4227 , piece_id=M5IKNOEODPJ5LLTCSP7ETVYPJIK25IEI3N4GITVD7ZVMMCBKMVMQ, seg_pos={0 0}, upload=57889]: Done uploading piece. err: <nil> Jul 26 23:59:05.613604659 (60h23m36.803903232s) [noise=true , address=77.90.31.13:4626 , piece_id=RJJ5RVRF37T62WN6S7ORB7JEUFYAOVBWSTFOFWVKQUA6ZSC6TXBQ, seg_pos={0 0}, upload=57889]: Done uploading piece. err: <nil> Jul 26 23:59:06.493944382 (60h23m37.684242973s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMVL64RDMPBHPUBOULZA75U4OSZEWVFXGJ6LZBW4NCRQC3NNVKA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: <nil> Jul 26 23:59:10.594392721 (60h23m41.784691301s) [noise=true , address=45.91.116.11:4119 , piece_id=QTT3QS6UPUDYOEJOLLAHGAOE5VS2576RQX66O63EL6TTQ35T3DTA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: rpc: dial tcp 45.91.116.11:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594460512 (60h23m41.784759091s) [noise=true , address=45.91.116.9:4263 , piece_id=HA43UJWPUMYJVV3DPS5IZTXCXOIFGR5JI3V6QIC6DOFX2AQ6YQFA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: rpc: dial tcp 45.91.116.9:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594496861 (60h23m41.784795441s) [noise=true , address=77.90.31.17:4770 , piece_id=6VV45BQLIEMG7YMYMCN5YWRZDDZN3QBGQ3JDTF3GWSIDZVVVIM3Q, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594567739 (60h23m41.784866324s) [noise=true , address=45.91.116.10:4081 , piece_id=OOSCSLRMBNR2HGMLH6JCQFGMSN2K6MVQEP32MY7ZXBRS723A63FQ, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: rpc: dial tcp 45.91.116.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594630485 (60h23m41.784929063s) [noise=true , address=45.91.116.4:4940 , piece_id=JZ4PBGYFMTWV5ASCLPZJP26VIPZC7U2H3W5HL4CITAA2LC5HOGBQ, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594647336 (60h23m41.784945908s) [noise=true , address=77.90.31.18:4805 , piece_id=O26N35M2TLTJVWSHV5BBPA3QQK6LT3AXSD6CLZMH3UGYNSVELQKA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594699711 (60h23m41.78499829s) [noise=true , address=104.36.82.9:4005 , piece_id=HXZ2MEAADB4J3KYPQYY276NME4HYDAS3TWKAPL4RRKYU7Q4YYBIA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 139.59.144.22:47168->104.36.82.9:4005: use of closed network connection; write tcp 139.59.144.22:47168->104.36.82.9:4005: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47168->104.36.82.9:4005: use of closed network connection Jul 26 23:59:10.594746689 (60h23m41.785045262s) [noise=true , address=77.90.31.16:4735 , piece_id=CRUIQTI2X2KJUAMASHI2GCAEST4VSKBTGYWWLXURDDM2DY3DJ7TQ, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594752869 (60h23m41.785051447s) [noise=true , address=45.91.116.13:4153 , piece_id=YI5N7Y2OPSPVJ2DEERT3WZKEK6NOAGAJJLP7HDMKEIKWEGGEDATA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594766163 (60h23m41.785064736s) [noise=true , address=77.90.31.14:4662 , piece_id=FP2HKN3AWPQ34I2FYW5B377ZWNPCBATJCQJ7NYLGELZEAU4AY6VA, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: rpc: dial tcp 77.90.31.14:4662: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:10.594775423 (60h23m41.785073994s) [noise=true , address=104.36.82.2:4732 , piece_id=EJCSRTNPIYJQBOCQXT2PFFOFO4AMSBR6G67CPTB5KGWI7BN5YUKQ, seg_pos={0 0}, upload=57889]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 139.59.144.22:34614->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:34614->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34614->104.36.82.2:4732: use of closed network connection Jul 26 23:59:10.594805674 (60h23m41.785104244s) [seg_pos={0 0}, upload=57889]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:59:10.706292839 (60h23m41.89659141s) [upload=57889]: Done uploading segments... Jul 26 23:59:10.706320734 (60h23m41.896619314s) [upload=57889]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.114743309s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..XX....X....X.X..X........XX.X....X......X..X............X.] duration: 58.520364942s FAIL (2025-07-26 23:59:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 58.520382961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 940.948336ms FAIL (2025-07-26 23:59:03 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 940.982343ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..XXXXX..X] duration: 30.118948322s FAIL (2025-07-26 23:59:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.11898833s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...X.X.X..X..X] duration: 1m0.003669057s FAIL (2025-07-26 23:58:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0h21m50.103173687s) [upload=57868]: Got next segment. Inline: false Jul 26 23:57:18.912912368 (60h21m50.10321095s) [upload=57868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:18.912919625 (60h21m50.103218195s) [upload=57868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:19.188683950 (60h21m50.378982538s) [seg_pos={0 0}, upload=57868]: Begin upload segment... Jul 26 23:57:19.188756380 (60h21m50.379054964s) [seg_pos={0 0}, upload=57868]: Done begin upload segment. Jul 26 23:57:19.188807003 (60h21m50.37910559s) [noise=true , address=79.127.201.215:4359 , piece_id=HE6LTNW7JEE6LI56XLYB4VNHSHCYUUSJZXFM2AOWGETVAZ4MUBCA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.189534789 (60h21m50.379833379s) [noise=true , address=45.91.116.9:4273 , piece_id=YOEO5I6NQEUCX3TTTIFM6ZMLDSY5JNGLZVOC2KW3LD3S4TA45XHQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.190043804 (60h21m50.380342396s) [noise=true , address=77.90.31.13:4658 , piece_id=27ZWPIB5SYGG6FCU6AXDUYWMFSVBAPJKAPQCC6D6MUJKDW67SC4Q, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.190085519 (60h21m50.380384109s) [noise=true , address=104.36.82.47:4528 , piece_id=FZCZ3QVXDYG5WORN4WU7O2XVTYOAGK4BT2MREIWH2JJJ57KGRP4A, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.190152196 (60h21m50.380450785s) [noise=true , address=104.36.82.46:4502 , piece_id=IGF5P45D4N6OPYIDEDEPSDTMKMHVRQA4JVNO523JKQIEGTSTYNLQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.190581750 (60h21m50.380880339s) [noise=true , address=77.90.31.10:4533 , piece_id=FSOBS44QAS75DV7LCDU7EXOQSNLJB26URH4U7UYW5O6FLYD75N5Q, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.190619407 (60h21m50.380917996s) [noise=true , address=104.36.82.53:4134 , piece_id=RV3BBFVFENAJCQI64BHLXAIX6MOUM6BEJI5SGN27PUV2DRAYIAFQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.191040185 (60h21m50.381338775s) [noise=true , address=79.127.205.235:4678 , piece_id=53TQSLK2KONUJNF345TQW4WB3KPBLPYPEXQ2PXGYL5D63MTZMB4Q, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.191072973 (60h21m50.381371562s) [noise=true , address=45.91.116.14:4206 , piece_id=GZLCCNJXVYDZXRL2IZ7LZIOWVPFHUANZLAEGSYDVKKK3QB2YR2AA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.191418159 (60h21m50.38171674s) [noise=true , address=104.36.82.13:4218 , piece_id=MB2WLIHMYEJSJQSJPGK4P4LQDSNEP5S75Y2T25JEKMHL4IDDGL5A, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.191523215 (60h21m50.381821806s) [noise=true , address=104.36.82.52:4119 , piece_id=UUOWWGKTHCZII57AP3UUBTAYFNIPY5A75MCFMSJ5XUETLGZTAX2A, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.191714471 (60h21m50.38201305s) [noise=true , address=77.90.31.11:4574 , piece_id=W5HBFJHFAAYKFRA5NHV7ZRTQQEASMFKLUBVRK3G7PJAJWSQV36VQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.191981670 (60h21m50.382280278s) [noise=true , address=79.127.205.233:4647 , piece_id=WM32PWTKAZPXUUM7567ZQW726ZNMAC4K2JHQLX67YQBSR5435QVQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.192015803 (60h21m50.382314382s) [noise=true , address=64.49.13.138:4483 , piece_id=3SCIGBS6YVIGUCVL65YNZDXI5WM3YHYPRSCF2OOSOV33JYKDC2EA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.192407733 (60h21m50.382706318s) [noise=true , address=64.49.13.140:4514 , piece_id=EOQZ2YA2EET67LLDT7WLAILLRGMNIXWWPXZOY5AGD26577LAWDXQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.192770473 (60h21m50.383069053s) [noise=true , address=45.91.116.10:4093 , piece_id=73E35XAHMHRD2VKIIYLTGNML75YNGIG2NOHRKPUOAMSCXCARJXGA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.193338291 (60h21m50.383636881s) [noise=true , address=79.127.205.234:4604 , piece_id=VR5J7V7QEBKU4KGBDOJXWZ4I2UA2A3XVRVFSHPOXDCBGCNPCWIHA, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.193689780 (60h21m50.383988353s) [noise=true , address=79.127.205.245:4250 , piece_id=VFLQMVCILKKKYM3ML5ZXZSBIDQ3KZNNCL5NGB3LL3PLY33X77UXQ, seg_pos={0 0}, upload=57868]: Uploading piece... Jul 26 23:57:19.764643793 (60h21m50.954942384s) [upload=57868]: Next returned nil segment Jul 26 23:57:19.764667333 (60h21m50.954965926s) [upload=57868]: Waiting for error group managing segments... Jul 26 23:57:19.947383626 (60h21m51.137682212s) [noise=true , address=77.90.31.11:4574 , piece_id=W5HBFJHFAAYKFRA5NHV7ZRTQQEASMFKLUBVRK3G7PJAJWSQV36VQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:20.443678621 (60h21m51.633977204s) [noise=true , address=79.127.205.235:4678 , piece_id=53TQSLK2KONUJNF345TQW4WB3KPBLPYPEXQ2PXGYL5D63MTZMB4Q, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:20.459223710 (60h21m51.649522289s) [noise=true , address=79.127.205.245:4250 , piece_id=VFLQMVCILKKKYM3ML5ZXZSBIDQ3KZNNCL5NGB3LL3PLY33X77UXQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:20.488921063 (60h21m51.679219636s) [noise=true , address=79.127.205.234:4604 , piece_id=VR5J7V7QEBKU4KGBDOJXWZ4I2UA2A3XVRVFSHPOXDCBGCNPCWIHA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:20.546053804 (60h21m51.736352378s) [noise=true , address=64.49.13.140:4514 , piece_id=EOQZ2YA2EET67LLDT7WLAILLRGMNIXWWPXZOY5AGD26577LAWDXQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:20.614698052 (60h21m51.804996619s) [noise=true , address=79.127.205.233:4647 , piece_id=WM32PWTKAZPXUUM7567ZQW726ZNMAC4K2JHQLX67YQBSR5435QVQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:21.699506513 (60h21m52.889805102s) [noise=true , address=64.49.13.138:4483 , piece_id=3SCIGBS6YVIGUCVL65YNZDXI5WM3YHYPRSCF2OOSOV33JYKDC2EA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:57:23.700089218 (60h21m54.890387791s) [noise=true , address=45.91.116.14:4206 , piece_id=GZLCCNJXVYDZXRL2IZ7LZIOWVPFHUANZLAEGSYDVKKK3QB2YR2AA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:26.031139026 (60h21m57.221437601s) [noise=true , address=77.90.31.13:4658 , piece_id=27ZWPIB5SYGG6FCU6AXDUYWMFSVBAPJKAPQCC6D6MUJKDW67SC4Q, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:27.163082135 (60h21m58.353380715s) [noise=true , address=45.91.116.10:4093 , piece_id=73E35XAHMHRD2VKIIYLTGNML75YNGIG2NOHRKPUOAMSCXCARJXGA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:28.425957634 (60h21m59.616256203s) [noise=true , address=45.91.116.9:4273 , piece_id=YOEO5I6NQEUCX3TTTIFM6ZMLDSY5JNGLZVOC2KW3LD3S4TA45XHQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:37.782335595 (60h22m8.97263417s) [noise=true , address=104.36.82.52:4119 , piece_id=UUOWWGKTHCZII57AP3UUBTAYFNIPY5A75MCFMSJ5XUETLGZTAX2A, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:51.372994030 (60h22m22.563292601s) [noise=true , address=79.127.201.215:4359 , piece_id=HE6LTNW7JEE6LI56XLYB4VNHSHCYUUSJZXFM2AOWGETVAZ4MUBCA, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:57:51.498573458 (60h22m22.688872032s) [noise=true , address=104.36.82.53:4134 , piece_id=RV3BBFVFENAJCQI64BHLXAIX6MOUM6BEJI5SGN27PUV2DRAYIAFQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:58:04.484000193 (60h22m35.674298759s) [noise=true , address=104.36.82.47:4528 , piece_id=FZCZ3QVXDYG5WORN4WU7O2XVTYOAGK4BT2MREIWH2JJJ57KGRP4A, seg_pos={0 0}, upload=57868]: Done uploading piece. err: <nil> Jul 26 23:58:04.484033390 (60h22m35.674331969s) [seg_pos={0 0}, upload=57868]: Segment reached optimal threshold of 14 pieces. Jul 26 23:58:04.484241323 (60h22m35.674539892s) [noise=true , address=104.36.82.46:4502 , piece_id=IGF5P45D4N6OPYIDEDEPSDTMKMHVRQA4JVNO523JKQIEGTSTYNLQ, seg_pos={0 0}, upload=57868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jul 26 23:58:04.484358102 (60h22m35.674656667s) [noise=true , address=77.90.31.10:4533 , piece_id=FSOBS44QAS75DV7LCDU7EXOQSNLJB26URH4U7UYW5O6FLYD75N5Q, seg_pos={0 0}, upload=57868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS): protocol: rpc: dial tcp 77.90.31.10:4533: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:04.484407138 (60h22m35.674705708s) [noise=true , address=104.36.82.13:4218 , piece_id=MB2WLIHMYEJSJQSJPGK4P4LQDSNEP5S75Y2T25JEKMHL4IDDGL5A, seg_pos={0 0}, upload=57868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: write tcp 139.59.144.22:52738->104.36.82.13:4218: use of closed network connection; write tcp 139.59.144.22:52738->104.36.82.13:4218: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52738->104.36.82.13:4218: use of closed network connection Jul 26 23:58:04.484431645 (60h22m35.674730213s) [seg_pos={0 0}, upload=57868]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 23:58:04.484523503 (60h22m35.674822073s) [upload=57868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:58:04.484529164 (60h22m35.674827743s) [upload=57868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:58:04.623011288 (60h22m35.813309873s) [upload=57868]: Done uploading segments... Jul 26 23:58:04.623066026 (60h22m35.813364611s) [upload=57868]: Upload finished. err: <nil> Duration: 1m0.003686843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...XX....X....X.X..X........XX.X....X......X..X............X] duration: 1m0.001514102s FAIL (2025-07-26 23:58:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001536895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.XX.XXXX] duration: 30.127528993s FAIL (2025-07-26 23:58:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== (60h22m4.411531025s) [upload=57874]: Got next segment. Inline: false Jul 26 23:57:33.221284815 (60h22m4.411583387s) [upload=57874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:57:33.221294858 (60h22m4.411593437s) [upload=57874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:57:33.330720396 (60h22m4.521018978s) [seg_pos={0 0}, upload=57874]: Begin upload segment... Jul 26 23:57:33.330768500 (60h22m4.521067071s) [seg_pos={0 0}, upload=57874]: Done begin upload segment. Jul 26 23:57:33.330811932 (60h22m4.521110524s) [noise=true , address=45.91.116.7:4010 , piece_id=FE5OO2SVJGITDZMZHLRRRMIXONBNGOSRWQG2IGIKJ2BI2LDHVWTQ, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.330863038 (60h22m4.52116163s) [noise=true , address=104.36.82.9:4006 , piece_id=URC7KOB5KAY6YAEKJEVGVDX56HVE2VWY7OICZOKWKOTTCSTBCR6A, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.331292060 (60h22m4.521590641s) [noise=true , address=45.91.116.10:4083 , piece_id=OVHMH2UXA7JS7HHOYAAXTF2VWLDNONVFMJDQCMR3LTHYIK6WNGWQ, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.331324337 (60h22m4.521622928s) [noise=true , address=104.36.82.8:4969 , piece_id=ZDITTMEJHZQY7HAV6MZSQWPHP22IIBS6DI4JF2EY2UNZT3CWXU2A, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.331731786 (60h22m4.522030373s) [noise=true , address=45.91.116.12:4225 , piece_id=PYVP4TNNUWPXPUHFRYSRCTUH4556F45Z6I4NFKSRBQO3V6CJDAIA, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.331749564 (60h22m4.522048153s) [noise=true , address=104.36.82.5:4838 , piece_id=NGI27ZUZI4HYQXKLVTOL5WW62P5YZZHQGWT66Y3MD6XDLZC4634Q, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.332207766 (60h22m4.522506359s) [noise=true , address=77.90.31.14:4662 , piece_id=3S5ZF7J6V3NNYZPCZXXTUK2SC3GLJ5S7TXOPR7R5FWRRXG76TH6A, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.332233767 (60h22m4.522532357s) [noise=true , address=45.91.116.4:4938 , piece_id=3KOKMZQFBSECQNY7LK3GVIZ5YXS5DY4LTJEQQLERBDLC6ILBMMEA, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.332606003 (60h22m4.522904593s) [noise=true , address=104.36.82.13:4187 , piece_id=THBUB3KQQZHHF5TIWKYH7PR7QBYPK3O7TSJZHWERQEP3PI6BGPTA, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.332638691 (60h22m4.522937273s) [noise=true , address=104.36.82.7:4909 , piece_id=SED4HBSPH32Z4SD3D5K3FPXMQBWAIBUMV47NHZARWITOZL4GG4XQ, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.333128764 (60h22m4.523427353s) [noise=true , address=104.36.82.10:4077 , piece_id=YTLRAYEWNCCPFQ3VRN3QUZ776RSQ7V62HFMUU5N245DKL4ZTWEVQ, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.333169310 (60h22m4.523467892s) [noise=true , address=45.91.116.11:4119 , piece_id=63PKPDX5CDZ3KO2RA2YFACZGRLANTKI2E66TRAGZWGEAKTQBWFZQ, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.333543104 (60h22m4.523841683s) [noise=true , address=77.90.31.10:4518 , piece_id=UGIRT4AHI6J2OPKC6QJAJFW7WU53EFC2DWQTZ4GWBYWP4O77GSIA, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.333554851 (60h22m4.523853442s) [noise=true , address=45.91.116.6:4973 , piece_id=ETO3PA4TBAUK2U2E2WKPI3CJC65XTQT4WDUOGKY7QT2SMLRA4VBQ, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.333963129 (60h22m4.52426171s) [noise=true , address=45.91.116.8:4048 , piece_id=UDHAGBC4KM3NAZ3WCCKAYMTMBA37KVNRU5ECDDJWEG3BFZHQCX6A, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.337392109 (60h22m4.527690695s) [noise=true , address=104.36.82.12:4150 , piece_id=4BDK6PKL7ZH7DEI4ISWMCMCA743GCHQ6ETPBY4KNFSVFDKYVRVTA, seg_pos={0 0}, upload=57874]: Uploading piece... Jul 26 23:57:33.943709691 (60h22m5.134008271s) [upload=57874]: Next returned nil segment Jul 26 23:57:33.943730174 (60h22m5.134028757s) [upload=57874]: Waiting for error group managing segments... Jul 26 23:57:34.081995068 (60h22m5.27229364s) [noise=true , address=45.91.116.11:4119 , piece_id=63PKPDX5CDZ3KO2RA2YFACZGRLANTKI2E66TRAGZWGEAKTQBWFZQ, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:57:34.083277732 (60h22m5.2735763s) [noise=true , address=45.91.116.4:4938 , piece_id=3KOKMZQFBSECQNY7LK3GVIZ5YXS5DY4LTJEQQLERBDLC6ILBMMEA, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:57:34.094417603 (60h22m5.284716182s) [noise=true , address=45.91.116.6:4973 , piece_id=ETO3PA4TBAUK2U2E2WKPI3CJC65XTQT4WDUOGKY7QT2SMLRA4VBQ, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:57:34.095130766 (60h22m5.285429331s) [noise=true , address=45.91.116.7:4010 , piece_id=FE5OO2SVJGITDZMZHLRRRMIXONBNGOSRWQG2IGIKJ2BI2LDHVWTQ, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:57:34.098266334 (60h22m5.288564907s) [noise=true , address=77.90.31.14:4662 , piece_id=3S5ZF7J6V3NNYZPCZXXTUK2SC3GLJ5S7TXOPR7R5FWRRXG76TH6A, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:57:34.099589121 (60h22m5.289887693s) [noise=true , address=45.91.116.12:4225 , piece_id=PYVP4TNNUWPXPUHFRYSRCTUH4556F45Z6I4NFKSRBQO3V6CJDAIA, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:57:34.101878161 (60h22m5.292176741s) [noise=true , address=45.91.116.10:4083 , piece_id=OVHMH2UXA7JS7HHOYAAXTF2VWLDNONVFMJDQCMR3LTHYIK6WNGWQ, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:57:34.105334988 (60h22m5.295633561s) [noise=true , address=45.91.116.8:4048 , piece_id=UDHAGBC4KM3NAZ3WCCKAYMTMBA37KVNRU5ECDDJWEG3BFZHQCX6A, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:58:01.134407589 (60h22m32.324706168s) [noise=true , address=104.36.82.12:4150 , piece_id=4BDK6PKL7ZH7DEI4ISWMCMCA743GCHQ6ETPBY4KNFSVFDKYVRVTA, seg_pos={0 0}, upload=57874]: Done uploading piece. err: <nil> Jul 26 23:58:02.945471915 (60h22m34.135770496s) [noise=true , address=104.36.82.7:4909 , piece_id=SED4HBSPH32Z4SD3D5K3FPXMQBWAIBUMV47NHZARWITOZL4GG4XQ, seg_pos={0 0}, upload=57874]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:58:02.945627123 (60h22m34.135925706s) [noise=true , address=104.36.82.8:4969 , piece_id=ZDITTMEJHZQY7HAV6MZSQWPHP22IIBS6DI4JF2EY2UNZT3CWXU2A, seg_pos={0 0}, upload=57874]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:43730->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:43730->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43730->104.36.82.8:4969: use of closed network connection Jul 26 23:58:02.945748421 (60h22m34.136046991s) [noise=true , address=104.36.82.10:4077 , piece_id=YTLRAYEWNCCPFQ3VRN3QUZ776RSQ7V62HFMUU5N245DKL4ZTWEVQ, seg_pos={0 0}, upload=57874]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:58:02.945835341 (60h22m34.136133911s) [noise=true , address=104.36.82.5:4838 , piece_id=NGI27ZUZI4HYQXKLVTOL5WW62P5YZZHQGWT66Y3MD6XDLZC4634Q, seg_pos={0 0}, upload=57874]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 139.59.144.22:46424->104.36.82.5:4838: use of closed network connection; write tcp 139.59.144.22:46424->104.36.82.5:4838: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46424->104.36.82.5:4838: use of closed network connection Jul 26 23:58:02.945862385 (60h22m34.136160967s) [noise=true , address=77.90.31.10:4518 , piece_id=UGIRT4AHI6J2OPKC6QJAJFW7WU53EFC2DWQTZ4GWBYWP4O77GSIA, seg_pos={0 0}, upload=57874]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:41088->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:02.945971033 (60h22m34.136269614s) [noise=true , address=104.36.82.9:4006 , piece_id=URC7KOB5KAY6YAEKJEVGVDX56HVE2VWY7OICZOKWKOTTCSTBCR6A, seg_pos={0 0}, upload=57874]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 139.59.144.22:44280->104.36.82.9:4006: use of closed network connection; write tcp 139.59.144.22:44280->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44280->104.36.82.9:4006: use of closed network connection Jul 26 23:58:02.945983678 (60h22m34.136282249s) [noise=true , address=104.36.82.13:4187 , piece_id=THBUB3KQQZHHF5TIWKYH7PR7QBYPK3O7TSJZHWERQEP3PI6BGPTA, seg_pos={0 0}, upload=57874]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:44702->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:44702->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44702->104.36.82.13:4187: use of closed network connection Jul 26 23:58:02.946027110 (60h22m34.136325684s) [seg_pos={0 0}, upload=57874]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:58:03.070494436 (60h22m34.260793026s) [upload=57874]: Done uploading segments... Jul 26 23:58:03.070550381 (60h22m34.260848993s) [upload=57874]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.127551135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 828.144807ms FAIL (2025-07-26 23:57:55 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 828.177909ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..XXXXX..] duration: 4.296144706s FAIL (2025-07-26 23:57:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.296174684s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X.......] duration: 4.027615803s FAIL (2025-07-26 23:57:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.027649911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.XX.XXX] duration: 30.104873718s FAIL (2025-07-26 23:57:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.084709423 (60h21m34.275008011s) [noise=true , address=104.36.82.7:4909 , piece_id=7PLZU2UVHT23XEL6CFULGLOFEVAV6WWMW622XTZVUDXQWNDLDLVQ, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.085618144 (60h21m34.275916733s) [noise=true , address=77.90.31.21:4879 , piece_id=XFSIESQJYALP26FFMMRTDIJMOWMM45EDEOURPTMKK7CHILJNYBUQ, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.085865114 (60h21m34.2761637s) [noise=true , address=45.91.116.8:4047 , piece_id=TN3MHUUKLVNQL2EPBJSKBAND57RHR2XVGM7RIQ7Z7XK3DV3WX7GA, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.086731726 (60h21m34.277030309s) [noise=true , address=104.36.82.12:4149 , piece_id=FLHJTTZGV4JYQIKVUICUOD2NG6UZSCTVDOBMYHZ4N5HV2VK7O65Q, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.086888820 (60h21m34.277187406s) [noise=true , address=104.36.82.8:4969 , piece_id=BV5JYOFOZTH3FVAZKNQX6I2LITXSGAEZUCE22RHJS7N66JBT3X2Q, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.087447126 (60h21m34.277745725s) [noise=true , address=77.90.31.18:4807 , piece_id=EAGVO7ROEQLK45GVWSXU3T2BAJUNCXYEEZKVHIMULTM7MSEAZELQ, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.087861157 (60h21m34.278159745s) [noise=true , address=77.90.31.10:4518 , piece_id=3STJHRMSNXXNBLKLBZV43KIFUHFBFLFJBADUEJCNURGXP5C6OIQQ, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.088281934 (60h21m34.278580524s) [noise=true , address=77.90.31.16:4734 , piece_id=7Q3RUWQUGMOLM5GFOX3TKWXKJLIIUCMYALWAW57YURAFPPWMRKXA, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.088667568 (60h21m34.278966159s) [noise=true , address=104.36.82.15:4258 , piece_id=Z733OCSL2M45BMTJKKQ56IMDMTURIPX6NVWMIAWK7VOL6DESJG2Q, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.089097813 (60h21m34.279396419s) [noise=true , address=77.90.31.15:4699 , piece_id=ECXKC7R65EMPODQGL3V2YLHZZJMZNWK3KIEDZIYMBKX6YHKO45IA, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.090330668 (60h21m34.280629256s) [noise=true , address=77.90.31.12:4592 , piece_id=C5JMTGMHGU4RVO46E6SSXOZOA5K5CFZWQOXOXMIL6TU5JQM7L3EA, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.090684855 (60h21m34.28098344s) [noise=true , address=45.91.116.5:4299 , piece_id=SBGTT5FFH4IERG7KTLHAIUXCG7S7ZUC2EQNIQENV3TTS7MLBLAXA, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.091010286 (60h21m34.281308869s) [noise=true , address=104.36.82.4:4804 , piece_id=4G7RBWJ5NUKSV5RP7NGYM5PEDE5RUPDCZHQFA2GCRQHL4JLM2GTQ, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.091407984 (60h21m34.281706575s) [noise=true , address=104.36.82.3:4767 , piece_id=K53TR3J7LSXGABNPT75EOPLVJYSPEU36TDXYT64LVP7UAYQBXL7Q, seg_pos={0 0}, upload=57865]: Uploading piece... Jul 26 23:57:03.698207221 (60h21m34.8885058s) [upload=57865]: Next returned nil segment Jul 26 23:57:03.698229600 (60h21m34.888528182s) [upload=57865]: Waiting for error group managing segments... Jul 26 23:57:03.841201319 (60h21m35.031499899s) [noise=true , address=45.91.116.5:4299 , piece_id=SBGTT5FFH4IERG7KTLHAIUXCG7S7ZUC2EQNIQENV3TTS7MLBLAXA, seg_pos={0 0}, upload=57865]: Done uploading piece. err: <nil> Jul 26 23:57:03.841406848 (60h21m35.031705418s) [noise=true , address=45.91.116.8:4047 , piece_id=TN3MHUUKLVNQL2EPBJSKBAND57RHR2XVGM7RIQ7Z7XK3DV3WX7GA, seg_pos={0 0}, upload=57865]: Done uploading piece. err: <nil> Jul 26 23:57:03.841819191 (60h21m35.03211777s) [noise=true , address=45.91.116.6:4973 , piece_id=24G5IRKZCHCHLAFRZTVHBEZZUWVDKOUAJ43K7CKSISVKJ7POYEIA, seg_pos={0 0}, upload=57865]: Done uploading piece. err: <nil> Jul 26 23:57:03.871585602 (60h21m35.061884172s) [noise=true , address=77.90.31.15:4699 , piece_id=ECXKC7R65EMPODQGL3V2YLHZZJMZNWK3KIEDZIYMBKX6YHKO45IA, seg_pos={0 0}, upload=57865]: Done uploading piece. err: <nil> Jul 26 23:57:03.874425940 (60h21m35.064724513s) [noise=true , address=77.90.31.16:4734 , piece_id=7Q3RUWQUGMOLM5GFOX3TKWXKJLIIUCMYALWAW57YURAFPPWMRKXA, seg_pos={0 0}, upload=57865]: Done uploading piece. err: <nil> Jul 26 23:57:05.233830623 (60h21m36.424129192s) [noise=true , address=77.90.31.12:4592 , piece_id=C5JMTGMHGU4RVO46E6SSXOZOA5K5CFZWQOXOXMIL6TU5JQM7L3EA, seg_pos={0 0}, upload=57865]: Done uploading piece. err: <nil> Jul 26 23:57:05.287154426 (60h21m36.477453016s) [noise=true , address=77.90.31.13:4627 , piece_id=5MM72EOMUCXH765FOKCNUJASCZYTBUDWX4HGIJN5AKO54TYUYIRQ, seg_pos={0 0}, upload=57865]: Done uploading piece. err: <nil> Jul 26 23:57:32.706443652 (60h22m3.896742233s) [noise=true , address=104.36.82.4:4804 , piece_id=4G7RBWJ5NUKSV5RP7NGYM5PEDE5RUPDCZHQFA2GCRQHL4JLM2GTQ, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 139.59.144.22:60086->104.36.82.4:4804: use of closed network connection; write tcp 139.59.144.22:60086->104.36.82.4:4804: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60086->104.36.82.4:4804: use of closed network connection Jul 26 23:57:32.706692864 (60h22m3.896991443s) [noise=true , address=77.90.31.10:4518 , piece_id=3STJHRMSNXXNBLKLBZV43KIFUHFBFLFJBADUEJCNURGXP5C6OIQQ, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:32.706909527 (60h22m3.897208106s) [noise=true , address=77.90.31.18:4807 , piece_id=EAGVO7ROEQLK45GVWSXU3T2BAJUNCXYEEZKVHIMULTM7MSEAZELQ, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:32.707062707 (60h22m3.897361288s) [noise=true , address=104.36.82.7:4909 , piece_id=7PLZU2UVHT23XEL6CFULGLOFEVAV6WWMW622XTZVUDXQWNDLDLVQ, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:34600->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:34600->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34600->104.36.82.7:4909: use of closed network connection Jul 26 23:57:32.707367542 (60h22m3.897666121s) [noise=true , address=77.90.31.21:4879 , piece_id=XFSIESQJYALP26FFMMRTDIJMOWMM45EDEOURPTMKK7CHILJNYBUQ, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:57:32.707448473 (60h22m3.897747052s) [noise=true , address=104.36.82.12:4149 , piece_id=FLHJTTZGV4JYQIKVUICUOD2NG6UZSCTVDOBMYHZ4N5HV2VK7O65Q, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 139.59.144.22:55042->104.36.82.12:4149: use of closed network connection; write tcp 139.59.144.22:55042->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55042->104.36.82.12:4149: use of closed network connection Jul 26 23:57:32.707518305 (60h22m3.897816886s) [noise=true , address=104.36.82.15:4258 , piece_id=Z733OCSL2M45BMTJKKQ56IMDMTURIPX6NVWMIAWK7VOL6DESJG2Q, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 139.59.144.22:42690->104.36.82.15:4258: use of closed network connection; write tcp 139.59.144.22:42690->104.36.82.15:4258: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42690->104.36.82.15:4258: use of closed network connection Jul 26 23:57:32.707583456 (60h22m3.897882036s) [noise=true , address=104.36.82.3:4767 , piece_id=K53TR3J7LSXGABNPT75EOPLVJYSPEU36TDXYT64LVP7UAYQBXL7Q, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:37712->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:37712->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37712->104.36.82.3:4767: use of closed network connection Jul 26 23:57:32.707634433 (60h22m3.897933011s) [noise=true , address=104.36.82.8:4969 , piece_id=BV5JYOFOZTH3FVAZKNQX6I2LITXSGAEZUCE22RHJS7N66JBT3X2Q, seg_pos={0 0}, upload=57865]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 139.59.144.22:41108->104.36.82.8:4969: use of closed network connection; write tcp 139.59.144.22:41108->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41108->104.36.82.8:4969: use of closed network connection Jul 26 23:57:32.707664396 (60h22m3.897962985s) [seg_pos={0 0}, upload=57865]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:57:32.808788048 (60h22m3.999086633s) [upload=57865]: Done uploading segments... Jul 26 23:57:32.808840367 (60h22m3.999138946s) [upload=57865]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.104893933s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X......] duration: 2.656787936s FAIL (2025-07-26 23:57:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.656813004s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...X.X.X..X..] duration: 51.904169816s FAIL (2025-07-26 23:57:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 805]: Upload finished. err: <nil> Jul 26 23:54:28.096334555 (60h18m59.286633135s) [upload=57825]: Starting upload Jul 26 23:54:28.096427911 (60h18m59.286726491s) [upload=57825]: Done starting upload Jul 26 23:54:28.096702226 (60h18m59.287000803s) [upload=57825]: Uploading segments... Jul 26 23:54:28.096900836 (60h18m59.287199431s) [upload=57825]: Got next segment. Inline: false Jul 26 23:54:28.096971617 (60h18m59.287270184s) [upload=57825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:28.096977831 (60h18m59.287276416s) [upload=57825]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:28.299542702 (60h18m59.489841282s) [seg_pos={0 0}, upload=57825]: Begin upload segment... Jul 26 23:54:28.299611901 (60h18m59.489910481s) [seg_pos={0 0}, upload=57825]: Done begin upload segment. Jul 26 23:54:28.299640166 (60h18m59.489938751s) [noise=true , address=104.36.82.56:4248 , piece_id=FMU6WFXEBKQ3XJBFOLFGIR5M2LHJEJC2RY7OQP5C6357CDN4A2DA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300031224 (60h18m59.490329794s) [noise=true , address=104.36.82.53:4130 , piece_id=BCP7RP4BD356BYESTHDFHITTOVWFG3KEPFD5XS2UPETI2ZQ6HITQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300091270 (60h18m59.490389842s) [noise=true , address=79.127.201.215:4346 , piece_id=TKNXFBYJCRUTSBYYPDP7L257VYPLBAXTDBEPAZOADJAOIMUVTR4Q, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300162445 (60h18m59.490461018s) [noise=true , address=77.90.31.21:4896 , piece_id=AF23VOXCUMFHTLOFSL2JBKN4QN5I2JDD4TVFA4POVHI334SVV5IA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300234442 (60h18m59.490533012s) [noise=true , address=79.127.201.214:4321 , piece_id=R6UHK6O3NNQ3YZCMZYNUH6WG4EOYWZZXWE5WFMQEUX5UL4DBE2MA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300303517 (60h18m59.49060209s) [noise=true , address=77.90.31.12:4605 , piece_id=7C5CGCIG3WGX3XGYLZMLSHH5OVWBDKSBN7Q4WDFYYXSWKVTQ6KJA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300366333 (60h18m59.490664905s) [noise=true , address=79.127.205.246:4288 , piece_id=5TAXPEU2X6H65G5U3AOAVVJH37K27V6QWVEGNGZD54F4QS4OWH6A, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300435349 (60h18m59.490733922s) [noise=true , address=104.36.82.14:4245 , piece_id=UT265LPWEB2VH4BXSTN3F5USGLKS35CHV6HR4BQKFSZ3SNADHGWA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300481120 (60h18m59.490779692s) [noise=true , address=77.90.31.13:4659 , piece_id=RJZE7ASOXH3NU5B7SPLND5JM4P4MREVZTNJJFBEH7TQ7WXTEUPHA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300541313 (60h18m59.490839885s) [noise=true , address=104.36.82.47:4524 , piece_id=ALQM7USNHP4GYCA3FVGGFDIE3PUTAEWWMBXZZ25VGIXSQSBWBRCQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300604572 (60h18m59.490903145s) [noise=true , address=79.127.219.38:4006 , piece_id=XU7XJPZQB5VNCLOY3PYAIZMCJCFUABLNJ3VQM7W2XY2WQ465A5OQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300662956 (60h18m59.490961528s) [noise=true , address=104.36.82.54:4179 , piece_id=3DCT2B3O6TG77U3G4PS3MDL2Q4ONN2BMFLOKTQBLEN4ULYJLIF4Q, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300709091 (60h18m59.491007663s) [noise=true , address=104.36.82.11:4132 , piece_id=FJCVE4MO4TSCZYXWZ5GJCUIDWE3OBDTDIVLMFUCER7JVY7GHBGGQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300765284 (60h18m59.491063856s) [noise=true , address=77.90.31.18:4832 , piece_id=HT5OBOQGBBBD2TXSFUH6XTWYZEI5ZYVPSCABAQPRCWA6SBLGKAWA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.301552953 (60h18m59.491851526s) [noise=true , address=77.90.31.17:4800 , piece_id=3KAJCF6WZ4LMBY6CTFQWVJP6U7DDIBSEO6L6TV4JJ3QHKVA3NPVA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.303412739 (60h18m59.493711311s) [noise=true , address=45.91.116.9:4273 , piece_id=4OQWWOIDPLPBXMHVYHFV4KZRT3IWZCISKQZJYDOITLOXUQRXCYWQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.303947199 (60h18m59.494245772s) [noise=true , address=77.90.31.19:4856 , piece_id=EDHQ5ASTQZN53D7MI6SYV43WYX5YUEBV6MJ6DID6UMLJDUWJDHHQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.304457267 (60h18m59.494755833s) [noise=true , address=79.127.205.235:4681 , piece_id=7MVAGJMGMG2OMUPNWY4JE67HR2GXRRY3LZ7BOGICMDCDNFVZP7XA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.880895881 (60h19m0.071194454s) [upload=57825]: Next returned nil segment Jul 26 23:54:28.880917029 (60h19m0.071215598s) [upload=57825]: Waiting for error group managing segments... Jul 26 23:54:28.938759139 (60h19m0.12905772s) [noise=true , address=79.127.201.214:4321 , piece_id=R6UHK6O3NNQ3YZCMZYNUH6WG4EOYWZZXWE5WFMQEUX5UL4DBE2MA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:29.022625594 (60h19m0.21292418s) [noise=true , address=45.91.116.9:4273 , piece_id=4OQWWOIDPLPBXMHVYHFV4KZRT3IWZCISKQZJYDOITLOXUQRXCYWQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.060699428 (60h19m0.250998001s) [noise=true , address=79.127.219.38:4006 , piece_id=XU7XJPZQB5VNCLOY3PYAIZMCJCFUABLNJ3VQM7W2XY2WQ465A5OQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.535800102 (60h19m0.726098682s) [noise=true , address=79.127.205.235:4681 , piece_id=7MVAGJMGMG2OMUPNWY4JE67HR2GXRRY3LZ7BOGICMDCDNFVZP7XA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.597980265 (60h19m0.78827883s) [noise=true , address=79.127.205.246:4288 , piece_id=5TAXPEU2X6H65G5U3AOAVVJH37K27V6QWVEGNGZD54F4QS4OWH6A, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:40.523474364 (60h19m11.713772929s) [noise=true , address=77.90.31.19:4856 , piece_id=EDHQ5ASTQZN53D7MI6SYV43WYX5YUEBV6MJ6DID6UMLJDUWJDHHQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:47.746672172 (60h19m18.936970744s) [noise=true , address=79.127.201.215:4346 , piece_id=TKNXFBYJCRUTSBYYPDP7L257VYPLBAXTDBEPAZOADJAOIMUVTR4Q, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:48.228273050 (60h19m19.418571622s) [noise=true , address=104.36.82.53:4130 , piece_id=BCP7RP4BD356BYESTHDFHITTOVWFG3KEPFD5XS2UPETI2ZQ6HITQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:51.426042821 (60h19m22.616341394s) [noise=true , address=77.90.31.12:4605 , piece_id=7C5CGCIG3WGX3XGYLZMLSHH5OVWBDKSBN7Q4WDFYYXSWKVTQ6KJA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:54.360685494 (60h19m25.550984067s) [noise=true , address=104.36.82.56:4248 , piece_id=FMU6WFXEBKQ3XJBFOLFGIR5M2LHJEJC2RY7OQP5C6357CDN4A2DA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:55.549356195 (60h19m26.739654764s) [noise=true , address=77.90.31.13:4659 , piece_id=RJZE7ASOXH3NU5B7SPLND5JM4P4MREVZTNJJFBEH7TQ7WXTEUPHA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:58.159243977 (60h19m29.349542555s) [noise=true , address=104.36.82.54:4179 , piece_id=3DCT2B3O6TG77U3G4PS3MDL2Q4ONN2BMFLOKTQBLEN4ULYJLIF4Q, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:06.621983856 (60h19m37.812282426s) [noise=true , address=104.36.82.47:4524 , piece_id=ALQM7USNHP4GYCA3FVGGFDIE3PUTAEWWMBXZZ25VGIXSQSBWBRCQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:24.918176111 (60h19m56.108474678s) [noise=true , address=104.36.82.11:4132 , piece_id=FJCVE4MO4TSCZYXWZ5GJCUIDWE3OBDTDIVLMFUCER7JVY7GHBGGQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:27.822461227 (60h19m59.012759806s) [noise=true , address=77.90.31.21:4896 , piece_id=AF23VOXCUMFHTLOFSL2JBKN4QN5I2JDD4TVFA4POVHI334SVV5IA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:27.822573319 (60h19m59.012871899s) [noise=true , address=104.36.82.14:4245 , piece_id=UT265LPWEB2VH4BXSTN3F5USGLKS35CHV6HR4BQKFSZ3SNADHGWA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:27.822629848 (60h19m59.012928429s) [noise=true , address=77.90.31.17:4800 , piece_id=3KAJCF6WZ4LMBY6CTFQWVJP6U7DDIBSEO6L6TV4JJ3QHKVA3NPVA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: rpc: dial tcp 77.90.31.17:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:27.822650434 (60h19m59.012949014s) [noise=true , address=77.90.31.18:4832 , piece_id=HT5OBOQGBBBD2TXSFUH6XTWYZEI5ZYVPSCABAQPRCWA6SBLGKAWA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 139.59.144.22:50530->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:27.822698989 (60h19m59.012997568s) [seg_pos={0 0}, upload=57825]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:55:27.950123107 (60h19m59.140421672s) [upload=57825]: Done uploading segments... Jul 26 23:55:27.950146794 (60h19m59.140445372s) [upload=57825]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 51.904194142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X.....] duration: 4.349360242s FAIL (2025-07-26 23:57:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.349391791s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [....XX....X....X.X..X........XX.X....X......X..X............] duration: 27.542548563s FAIL (2025-07-26 23:57:07 UTC) orbiter warning: context deadline exceeded Duration: 27.542568151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.XX.XX] duration: 30.11432247s FAIL (2025-07-26 23:57:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== cmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:50876->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:50876->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50876->104.36.82.11:4113: use of closed network connection Jul 26 23:55:53.802905643 (60h20m24.993204225s) [seg_pos={0 0}, upload=57838]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:55:53.927919871 (60h20m25.118218437s) [upload=57838]: Done uploading segments... Jul 26 23:55:53.927945873 (60h20m25.118244453s) [upload=57838]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:56:32.716004186 (60h21m3.906302763s) [upload=57856]: Starting upload Jul 26 23:56:32.716112673 (60h21m3.906411258s) [upload=57856]: Done starting upload Jul 26 23:56:32.716391386 (60h21m3.906689956s) [upload=57856]: Uploading segments... Jul 26 23:56:32.716440443 (60h21m3.906739031s) [upload=57856]: Got next segment. Inline: false Jul 26 23:56:32.716499436 (60h21m3.906798015s) [upload=57856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:56:32.716505206 (60h21m3.906803788s) [upload=57856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:56:32.823745516 (60h21m4.014044105s) [seg_pos={0 0}, upload=57856]: Begin upload segment... Jul 26 23:56:32.823830298 (60h21m4.014128877s) [seg_pos={0 0}, upload=57856]: Done begin upload segment. Jul 26 23:56:32.823888205 (60h21m4.014186795s) [noise=true , address=77.90.31.10:4517 , piece_id=TZHBABTYEXAQXB3YE6AANWNSAX54P5UXIJTCUPG6GDYPFCQHQJKA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.823993167 (60h21m4.014291756s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQ4ZYTKBX5FZFL36HFER4N7SVR7DG5Z2WXYTAAATMJ2QDTSN7JLA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.825116473 (60h21m4.01541506s) [noise=true , address=45.91.116.5:4299 , piece_id=WH2VO6S3HVYDW3XEYALFLI5VWHBN5GGWB6GMC5ALXDNWTOR4XBXQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.825248145 (60h21m4.01554673s) [noise=true , address=45.91.116.14:4191 , piece_id=U37YBKANHWYYX3GLXRS563KVP6DLT5R7TTUYE6OWR7VOG2SBZ3GA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.825566361 (60h21m4.015864943s) [noise=true , address=45.91.116.9:4261 , piece_id=P2EIOFLPCOUAWKPW6SL5DAETD5N65LQFPGDLFFNLBLQWHFVSLDXQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.825661081 (60h21m4.015959662s) [noise=true , address=45.91.116.11:4119 , piece_id=GBE6WHGDEN33PQPHICRWJB27DUUHBZPHPGV3JZBEHJLEG7SX6GHA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.825995623 (60h21m4.016294209s) [noise=true , address=45.91.116.13:4155 , piece_id=SYU5ZNN5TGMSZFM7ICB6KGS4VHV5XEEEJVB3YB4QYDBYZ35EO3WA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.826037004 (60h21m4.016335584s) [noise=true , address=104.36.82.5:4839 , piece_id=L7QUF7S6LSVEOHIDUBYN5MLEIRIZQWFVFFOQUIOPC5ASLPPETDEQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.826380789 (60h21m4.016679378s) [noise=true , address=45.91.116.12:4225 , piece_id=EAXKOGKAWJL4W3E62ZAA2HG3KASLMZZNGBIN2NQJFJXDRM6PLKEA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.826399302 (60h21m4.016697884s) [noise=true , address=104.36.82.11:4114 , piece_id=M4PSUHHAV37P57RQB2HHTKABVGT52ZLCYC4UYGDAWJPKVO4XT6SA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.826768745 (60h21m4.017067341s) [noise=true , address=45.91.116.11:4118 , piece_id=F5V4L2IBRCYWUM237CCGIJMCPMVZXZPOLH7M7INTG2CQXJQ6MGOA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.826782256 (60h21m4.017080835s) [noise=true , address=104.36.82.8:4971 , piece_id=N27EWUTRTIMEJ747N2HLYK7LVM5UIA2R6UDTBRHEM7B4FM5PXUPQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.827095499 (60h21m4.017394078s) [noise=true , address=45.91.116.7:4009 , piece_id=SEBMJPEBTGTTGVVUW3HWYRYHDBIHN3IQEY7XUV3XWBCL6IPEFNRQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.827142004 (60h21m4.017440583s) [noise=true , address=45.91.116.6:4974 , piece_id=DVMBKCSYHCDFNMZYCJPDMNHDINJYG4CG2ZMKKDX2UW43W5CSHBWA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.827371619 (60h21m4.017670201s) [noise=true , address=104.36.82.12:4149 , piece_id=5QSBIW2Q4LS3WHTUBEFS3K7H525Y7XBOZAPJLHA4SV4M4RDHA4SA, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:32.829746255 (60h21m4.020044842s) [noise=true , address=77.90.31.12:4590 , piece_id=M6H55A6SUSRYNRW76KIPEUAAWWZI66PEZRSRVICNW3PONRY2A6DQ, seg_pos={0 0}, upload=57856]: Uploading piece... Jul 26 23:56:33.438978041 (60h21m4.629276611s) [upload=57856]: Next returned nil segment Jul 26 23:56:33.438996339 (60h21m4.629294909s) [upload=57856]: Waiting for error group managing segments... Jul 26 23:56:33.570350052 (60h21m4.760648638s) [noise=true , address=45.91.116.9:4261 , piece_id=P2EIOFLPCOUAWKPW6SL5DAETD5N65LQFPGDLFFNLBLQWHFVSLDXQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.572184624 (60h21m4.762483196s) [noise=true , address=45.91.116.11:4119 , piece_id=GBE6WHGDEN33PQPHICRWJB27DUUHBZPHPGV3JZBEHJLEG7SX6GHA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.574988802 (60h21m4.76528738s) [noise=true , address=45.91.116.11:4118 , piece_id=F5V4L2IBRCYWUM237CCGIJMCPMVZXZPOLH7M7INTG2CQXJQ6MGOA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.575025274 (60h21m4.765323862s) [noise=true , address=45.91.116.12:4225 , piece_id=EAXKOGKAWJL4W3E62ZAA2HG3KASLMZZNGBIN2NQJFJXDRM6PLKEA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.575548047 (60h21m4.765846624s) [noise=true , address=45.91.116.14:4191 , piece_id=U37YBKANHWYYX3GLXRS563KVP6DLT5R7TTUYE6OWR7VOG2SBZ3GA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.581556108 (60h21m4.771854687s) [noise=true , address=45.91.116.7:4009 , piece_id=SEBMJPEBTGTTGVVUW3HWYRYHDBIHN3IQEY7XUV3XWBCL6IPEFNRQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.590895873 (60h21m4.781194441s) [noise=true , address=45.91.116.6:4974 , piece_id=DVMBKCSYHCDFNMZYCJPDMNHDINJYG4CG2ZMKKDX2UW43W5CSHBWA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.602041775 (60h21m4.79234034s) [noise=true , address=45.91.116.13:4155 , piece_id=SYU5ZNN5TGMSZFM7ICB6KGS4VHV5XEEEJVB3YB4QYDBYZ35EO3WA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:33.605588587 (60h21m4.795887175s) [noise=true , address=45.91.116.5:4299 , piece_id=WH2VO6S3HVYDW3XEYALFLI5VWHBN5GGWB6GMC5ALXDNWTOR4XBXQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:34.640876920 (60h21m5.831175506s) [noise=true , address=77.90.31.10:4517 , piece_id=TZHBABTYEXAQXB3YE6AANWNSAX54P5UXIJTCUPG6GDYPFCQHQJKA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:35.598778439 (60h21m6.789077016s) [noise=true , address=77.90.31.12:4590 , piece_id=M6H55A6SUSRYNRW76KIPEUAAWWZI66PEZRSRVICNW3PONRY2A6DQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:56:45.232649395 (60h21m16.422947963s) [noise=true , address=104.36.82.11:4114 , piece_id=M4PSUHHAV37P57RQB2HHTKABVGT52ZLCYC4UYGDAWJPKVO4XT6SA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: <nil> Jul 26 23:57:02.440478620 (60h21m33.630777209s) [noise=true , address=104.36.82.8:4971 , piece_id=N27EWUTRTIMEJ747N2HLYK7LVM5UIA2R6UDTBRHEM7B4FM5PXUPQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:47934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:47934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47934->104.36.82.8:4971: use of closed network connection Jul 26 23:57:02.440508840 (60h21m33.630807422s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQ4ZYTKBX5FZFL36HFER4N7SVR7DG5Z2WXYTAAATMJ2QDTSN7JLA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:57:02.440555272 (60h21m33.630853842s) [noise=true , address=104.36.82.5:4839 , piece_id=L7QUF7S6LSVEOHIDUBYN5MLEIRIZQWFVFFOQUIOPC5ASLPPETDEQ, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:39196->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:39196->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39196->104.36.82.5:4839: use of closed network connection Jul 26 23:57:02.440744155 (60h21m33.631042743s) [noise=true , address=104.36.82.12:4149 , piece_id=5QSBIW2Q4LS3WHTUBEFS3K7H525Y7XBOZAPJLHA4SV4M4RDHA4SA, seg_pos={0 0}, upload=57856]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: read tcp 139.59.144.22:34192->104.36.82.12:4149: use of closed network connection; context deadline exceeded Jul 26 23:57:02.440796692 (60h21m33.631095282s) [seg_pos={0 0}, upload=57856]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:57:02.553363143 (60h21m33.743661715s) [upload=57856]: Done uploading segments... Jul 26 23:57:02.553388098 (60h21m33.743686678s) [upload=57856]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.114412331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..XXXXX.] duration: 2.435457535s FAIL (2025-07-26 23:57:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.435495015s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 827.955728ms FAIL (2025-07-26 23:56:52 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 827.994082ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X....] duration: 2.709746832s FAIL (2025-07-26 23:56:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.709761382s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X...] duration: 19.000804754s FAIL (2025-07-26 23:56:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.000831388s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.....XX....X....X.X..X........XX.X....X......X..X...........] duration: 36.639942634s FAIL (2025-07-26 23:56:39 UTC) orbiter warning: context deadline exceeded Duration: 36.639978003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..XXXXX] duration: 30.107282485s FAIL (2025-07-26 23:56:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.107330426s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...X.X.X..X.] duration: 58.328327384s FAIL (2025-07-26 23:56:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 805]: Upload finished. err: <nil> Jul 26 23:54:28.096334555 (60h18m59.286633135s) [upload=57825]: Starting upload Jul 26 23:54:28.096427911 (60h18m59.286726491s) [upload=57825]: Done starting upload Jul 26 23:54:28.096702226 (60h18m59.287000803s) [upload=57825]: Uploading segments... Jul 26 23:54:28.096900836 (60h18m59.287199431s) [upload=57825]: Got next segment. Inline: false Jul 26 23:54:28.096971617 (60h18m59.287270184s) [upload=57825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:28.096977831 (60h18m59.287276416s) [upload=57825]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:28.299542702 (60h18m59.489841282s) [seg_pos={0 0}, upload=57825]: Begin upload segment... Jul 26 23:54:28.299611901 (60h18m59.489910481s) [seg_pos={0 0}, upload=57825]: Done begin upload segment. Jul 26 23:54:28.299640166 (60h18m59.489938751s) [noise=true , address=104.36.82.56:4248 , piece_id=FMU6WFXEBKQ3XJBFOLFGIR5M2LHJEJC2RY7OQP5C6357CDN4A2DA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300031224 (60h18m59.490329794s) [noise=true , address=104.36.82.53:4130 , piece_id=BCP7RP4BD356BYESTHDFHITTOVWFG3KEPFD5XS2UPETI2ZQ6HITQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300091270 (60h18m59.490389842s) [noise=true , address=79.127.201.215:4346 , piece_id=TKNXFBYJCRUTSBYYPDP7L257VYPLBAXTDBEPAZOADJAOIMUVTR4Q, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300162445 (60h18m59.490461018s) [noise=true , address=77.90.31.21:4896 , piece_id=AF23VOXCUMFHTLOFSL2JBKN4QN5I2JDD4TVFA4POVHI334SVV5IA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300234442 (60h18m59.490533012s) [noise=true , address=79.127.201.214:4321 , piece_id=R6UHK6O3NNQ3YZCMZYNUH6WG4EOYWZZXWE5WFMQEUX5UL4DBE2MA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300303517 (60h18m59.49060209s) [noise=true , address=77.90.31.12:4605 , piece_id=7C5CGCIG3WGX3XGYLZMLSHH5OVWBDKSBN7Q4WDFYYXSWKVTQ6KJA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300366333 (60h18m59.490664905s) [noise=true , address=79.127.205.246:4288 , piece_id=5TAXPEU2X6H65G5U3AOAVVJH37K27V6QWVEGNGZD54F4QS4OWH6A, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300435349 (60h18m59.490733922s) [noise=true , address=104.36.82.14:4245 , piece_id=UT265LPWEB2VH4BXSTN3F5USGLKS35CHV6HR4BQKFSZ3SNADHGWA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300481120 (60h18m59.490779692s) [noise=true , address=77.90.31.13:4659 , piece_id=RJZE7ASOXH3NU5B7SPLND5JM4P4MREVZTNJJFBEH7TQ7WXTEUPHA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300541313 (60h18m59.490839885s) [noise=true , address=104.36.82.47:4524 , piece_id=ALQM7USNHP4GYCA3FVGGFDIE3PUTAEWWMBXZZ25VGIXSQSBWBRCQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300604572 (60h18m59.490903145s) [noise=true , address=79.127.219.38:4006 , piece_id=XU7XJPZQB5VNCLOY3PYAIZMCJCFUABLNJ3VQM7W2XY2WQ465A5OQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300662956 (60h18m59.490961528s) [noise=true , address=104.36.82.54:4179 , piece_id=3DCT2B3O6TG77U3G4PS3MDL2Q4ONN2BMFLOKTQBLEN4ULYJLIF4Q, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300709091 (60h18m59.491007663s) [noise=true , address=104.36.82.11:4132 , piece_id=FJCVE4MO4TSCZYXWZ5GJCUIDWE3OBDTDIVLMFUCER7JVY7GHBGGQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300765284 (60h18m59.491063856s) [noise=true , address=77.90.31.18:4832 , piece_id=HT5OBOQGBBBD2TXSFUH6XTWYZEI5ZYVPSCABAQPRCWA6SBLGKAWA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.301552953 (60h18m59.491851526s) [noise=true , address=77.90.31.17:4800 , piece_id=3KAJCF6WZ4LMBY6CTFQWVJP6U7DDIBSEO6L6TV4JJ3QHKVA3NPVA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.303412739 (60h18m59.493711311s) [noise=true , address=45.91.116.9:4273 , piece_id=4OQWWOIDPLPBXMHVYHFV4KZRT3IWZCISKQZJYDOITLOXUQRXCYWQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.303947199 (60h18m59.494245772s) [noise=true , address=77.90.31.19:4856 , piece_id=EDHQ5ASTQZN53D7MI6SYV43WYX5YUEBV6MJ6DID6UMLJDUWJDHHQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.304457267 (60h18m59.494755833s) [noise=true , address=79.127.205.235:4681 , piece_id=7MVAGJMGMG2OMUPNWY4JE67HR2GXRRY3LZ7BOGICMDCDNFVZP7XA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.880895881 (60h19m0.071194454s) [upload=57825]: Next returned nil segment Jul 26 23:54:28.880917029 (60h19m0.071215598s) [upload=57825]: Waiting for error group managing segments... Jul 26 23:54:28.938759139 (60h19m0.12905772s) [noise=true , address=79.127.201.214:4321 , piece_id=R6UHK6O3NNQ3YZCMZYNUH6WG4EOYWZZXWE5WFMQEUX5UL4DBE2MA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:29.022625594 (60h19m0.21292418s) [noise=true , address=45.91.116.9:4273 , piece_id=4OQWWOIDPLPBXMHVYHFV4KZRT3IWZCISKQZJYDOITLOXUQRXCYWQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.060699428 (60h19m0.250998001s) [noise=true , address=79.127.219.38:4006 , piece_id=XU7XJPZQB5VNCLOY3PYAIZMCJCFUABLNJ3VQM7W2XY2WQ465A5OQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.535800102 (60h19m0.726098682s) [noise=true , address=79.127.205.235:4681 , piece_id=7MVAGJMGMG2OMUPNWY4JE67HR2GXRRY3LZ7BOGICMDCDNFVZP7XA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.597980265 (60h19m0.78827883s) [noise=true , address=79.127.205.246:4288 , piece_id=5TAXPEU2X6H65G5U3AOAVVJH37K27V6QWVEGNGZD54F4QS4OWH6A, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:40.523474364 (60h19m11.713772929s) [noise=true , address=77.90.31.19:4856 , piece_id=EDHQ5ASTQZN53D7MI6SYV43WYX5YUEBV6MJ6DID6UMLJDUWJDHHQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:47.746672172 (60h19m18.936970744s) [noise=true , address=79.127.201.215:4346 , piece_id=TKNXFBYJCRUTSBYYPDP7L257VYPLBAXTDBEPAZOADJAOIMUVTR4Q, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:48.228273050 (60h19m19.418571622s) [noise=true , address=104.36.82.53:4130 , piece_id=BCP7RP4BD356BYESTHDFHITTOVWFG3KEPFD5XS2UPETI2ZQ6HITQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:51.426042821 (60h19m22.616341394s) [noise=true , address=77.90.31.12:4605 , piece_id=7C5CGCIG3WGX3XGYLZMLSHH5OVWBDKSBN7Q4WDFYYXSWKVTQ6KJA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:54.360685494 (60h19m25.550984067s) [noise=true , address=104.36.82.56:4248 , piece_id=FMU6WFXEBKQ3XJBFOLFGIR5M2LHJEJC2RY7OQP5C6357CDN4A2DA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:55.549356195 (60h19m26.739654764s) [noise=true , address=77.90.31.13:4659 , piece_id=RJZE7ASOXH3NU5B7SPLND5JM4P4MREVZTNJJFBEH7TQ7WXTEUPHA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:58.159243977 (60h19m29.349542555s) [noise=true , address=104.36.82.54:4179 , piece_id=3DCT2B3O6TG77U3G4PS3MDL2Q4ONN2BMFLOKTQBLEN4ULYJLIF4Q, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:06.621983856 (60h19m37.812282426s) [noise=true , address=104.36.82.47:4524 , piece_id=ALQM7USNHP4GYCA3FVGGFDIE3PUTAEWWMBXZZ25VGIXSQSBWBRCQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:24.918176111 (60h19m56.108474678s) [noise=true , address=104.36.82.11:4132 , piece_id=FJCVE4MO4TSCZYXWZ5GJCUIDWE3OBDTDIVLMFUCER7JVY7GHBGGQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:27.822461227 (60h19m59.012759806s) [noise=true , address=77.90.31.21:4896 , piece_id=AF23VOXCUMFHTLOFSL2JBKN4QN5I2JDD4TVFA4POVHI334SVV5IA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:27.822573319 (60h19m59.012871899s) [noise=true , address=104.36.82.14:4245 , piece_id=UT265LPWEB2VH4BXSTN3F5USGLKS35CHV6HR4BQKFSZ3SNADHGWA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:27.822629848 (60h19m59.012928429s) [noise=true , address=77.90.31.17:4800 , piece_id=3KAJCF6WZ4LMBY6CTFQWVJP6U7DDIBSEO6L6TV4JJ3QHKVA3NPVA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: rpc: dial tcp 77.90.31.17:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:27.822650434 (60h19m59.012949014s) [noise=true , address=77.90.31.18:4832 , piece_id=HT5OBOQGBBBD2TXSFUH6XTWYZEI5ZYVPSCABAQPRCWA6SBLGKAWA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 139.59.144.22:50530->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:27.822698989 (60h19m59.012997568s) [seg_pos={0 0}, upload=57825]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:55:27.950123107 (60h19m59.140421672s) [upload=57825]: Done uploading segments... Jul 26 23:55:27.950146794 (60h19m59.140445372s) [upload=57825]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 58.328362266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 949.066132ms FAIL (2025-07-26 23:56:21 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 949.120742ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..XXXX] duration: 30.118289929s FAIL (2025-07-26 23:56:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.118313761s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.XX.X] duration: 30.128379685s FAIL (2025-07-26 23:55:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9.434053096s) [noise=true , address=104.36.82.11:4113 , piece_id=75AV4SAQLTREYJI24IIWTQE2HIPMQUFTJQJ2UAIWECKQYPZL6JOQ, seg_pos={0 0}, upload=57832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: context canceled; context canceled Jul 26 23:55:18.243769875 (60h19m49.434068457s) [seg_pos={0 0}, upload=57832]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:55:18.243883556 (60h19m49.434182135s) [upload=57832]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:55:18.243888535 (60h19m49.434187114s) [upload=57832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:55:18.423705076 (60h19m49.614003665s) [upload=57832]: Done uploading segments... Jul 26 23:55:18.423729156 (60h19m49.614027728s) [upload=57832]: Upload finished. err: <nil> Jul 26 23:55:24.071554244 (60h19m55.261852822s) [upload=57838]: Starting upload Jul 26 23:55:24.071634016 (60h19m55.261932596s) [upload=57838]: Done starting upload Jul 26 23:55:24.071715227 (60h19m55.262013793s) [upload=57838]: Uploading segments... Jul 26 23:55:24.072014350 (60h19m55.262312929s) [upload=57838]: Got next segment. Inline: false Jul 26 23:55:24.072046387 (60h19m55.262344966s) [upload=57838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:55:24.072052255 (60h19m55.26235082s) [upload=57838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:55:24.254097977 (60h19m55.444396566s) [seg_pos={0 0}, upload=57838]: Begin upload segment... Jul 26 23:55:24.254166247 (60h19m55.444464818s) [seg_pos={0 0}, upload=57838]: Done begin upload segment. Jul 26 23:55:24.254192431 (60h19m55.444491019s) [noise=true , address=104.36.82.7:4909 , piece_id=4PDUZPILW2VMGSEATSZ7IN6ADW272OXBPISNXMWAQ4IFDYOF7FMQ, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.254210552 (60h19m55.444509131s) [noise=true , address=45.91.116.12:4227 , piece_id=DROBNI3D2C6TGK2KFDRPUIZCUO3BWS7OHRKSZR5G5IKOA34INWKA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.254610233 (60h19m55.444908816s) [noise=true , address=77.90.31.13:4627 , piece_id=2TQVZCGRMCDKGQUQROY6OFU5IFTEJR3GA7KNFMG5HBLRC6ZJFYVQ, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.254623625 (60h19m55.444922207s) [noise=true , address=77.90.31.12:4590 , piece_id=U7LAP6RQRF5SH4KWUUIINFHIE2N5MMAPSKMZG4RGOOCSZFDEZMBQ, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.254935797 (60h19m55.445234377s) [noise=true , address=104.36.82.5:4839 , piece_id=3GV3PUJ2CLBNYWILAR7SYHJR7WPOKAZEOAUQV33HRZ3D76ASB5LA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.254961153 (60h19m55.445259736s) [noise=true , address=45.91.116.7:4009 , piece_id=W5GGEXBDWGOSY6JLMXGOSJZNLOGZXOUWUYFDUI4C4MGQT72CM6OQ, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.255253172 (60h19m55.445551753s) [noise=true , address=45.91.116.13:4154 , piece_id=I5UKOJKMT3HXGXHQORVRNTDXP3QQ7FH3LP7WIWOCXGHBSPZSWRQA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.255380380 (60h19m55.445679065s) [noise=true , address=45.91.116.6:4974 , piece_id=QYFSZVI37ATSMGOJQL4WZVJAFRCYK7Z22BHXA7OYL2737JZT77EA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.255589449 (60h19m55.445888031s) [noise=true , address=77.90.31.14:4663 , piece_id=64H2P26R65VDJVWEQ77N2SW2GUCJZU37QMYHFCA36TUMFKQBH6ZA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.255842001 (60h19m55.446140591s) [noise=true , address=77.90.31.17:4770 , piece_id=QPI7MIMEZBMM62UGJ4O27UMWKDLWQ5D7C26XZ6WLOWCXVEBEMNBA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.255877424 (60h19m55.446176007s) [noise=true , address=104.36.82.8:4972 , piece_id=EH2WQVULBSEWAKV7BHJH4K2QEPETGV5MJECTAVNMYYPHSMW5XJDQ, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.256173880 (60h19m55.446472462s) [noise=true , address=45.91.116.7:4010 , piece_id=CRBROL7DVS25AONQLHBOS5BFNIZMNFXR5ZKO4ZQ4OQ7UOXZKNVLQ, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.256178646 (60h19m55.446477228s) [noise=true , address=45.91.116.8:4045 , piece_id=I3YCKGQA6TTWSCVNWSEIPIRTHWKNRTURJYILHACVDCROW5Q4XFCA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.256469373 (60h19m55.446767953s) [noise=true , address=77.90.31.10:4518 , piece_id=CVMVI6MZXPBLUGXKGNGM6GEJXNN4BPDKJJUVNKEJDUVRWWN6QNNA, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.256519909 (60h19m55.446818494s) [noise=true , address=104.36.82.11:4113 , piece_id=FABIS2URHW6HOY6HKCZQT2LGILZVAZI4JFB42GBLYVLRCAH6WNDQ, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.256739967 (60h19m55.447038539s) [noise=true , address=45.91.116.10:4083 , piece_id=6BQ3YMSL5U2FPAMF3QJPGSRPJOLYVUVEU43ZPE42RUTC7YLANE7A, seg_pos={0 0}, upload=57838]: Uploading piece... Jul 26 23:55:24.862888024 (60h19m56.05318659s) [upload=57838]: Next returned nil segment Jul 26 23:55:24.862905002 (60h19m56.053203584s) [upload=57838]: Waiting for error group managing segments... Jul 26 23:55:25.002292840 (60h19m56.192591409s) [noise=true , address=45.91.116.8:4045 , piece_id=I3YCKGQA6TTWSCVNWSEIPIRTHWKNRTURJYILHACVDCROW5Q4XFCA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:25.003403170 (60h19m56.193701736s) [noise=true , address=45.91.116.13:4154 , piece_id=I5UKOJKMT3HXGXHQORVRNTDXP3QQ7FH3LP7WIWOCXGHBSPZSWRQA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:25.003468791 (60h19m56.19376737s) [noise=true , address=45.91.116.6:4974 , piece_id=QYFSZVI37ATSMGOJQL4WZVJAFRCYK7Z22BHXA7OYL2737JZT77EA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:25.005063521 (60h19m56.195362107s) [noise=true , address=45.91.116.7:4010 , piece_id=CRBROL7DVS25AONQLHBOS5BFNIZMNFXR5ZKO4ZQ4OQ7UOXZKNVLQ, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:25.007192045 (60h19m56.197490611s) [noise=true , address=45.91.116.10:4083 , piece_id=6BQ3YMSL5U2FPAMF3QJPGSRPJOLYVUVEU43ZPE42RUTC7YLANE7A, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:25.011627075 (60h19m56.201925647s) [noise=true , address=45.91.116.12:4227 , piece_id=DROBNI3D2C6TGK2KFDRPUIZCUO3BWS7OHRKSZR5G5IKOA34INWKA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:25.016654375 (60h19m56.20695294s) [noise=true , address=45.91.116.7:4009 , piece_id=W5GGEXBDWGOSY6JLMXGOSJZNLOGZXOUWUYFDUI4C4MGQT72CM6OQ, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:25.016787644 (60h19m56.207086219s) [noise=true , address=77.90.31.14:4663 , piece_id=64H2P26R65VDJVWEQ77N2SW2GUCJZU37QMYHFCA36TUMFKQBH6ZA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:26.418829453 (60h19m57.60912804s) [noise=true , address=77.90.31.13:4627 , piece_id=2TQVZCGRMCDKGQUQROY6OFU5IFTEJR3GA7KNFMG5HBLRC6ZJFYVQ, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:27.114910828 (60h19m58.305209396s) [noise=true , address=77.90.31.12:4590 , piece_id=U7LAP6RQRF5SH4KWUUIINFHIE2N5MMAPSKMZG4RGOOCSZFDEZMBQ, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:34.271968111 (60h20m5.462266692s) [noise=true , address=104.36.82.7:4909 , piece_id=4PDUZPILW2VMGSEATSZ7IN6ADW272OXBPISNXMWAQ4IFDYOF7FMQ, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:35.009318802 (60h20m6.199617372s) [noise=true , address=77.90.31.10:4518 , piece_id=CVMVI6MZXPBLUGXKGNGM6GEJXNN4BPDKJJUVNKEJDUVRWWN6QNNA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: <nil> Jul 26 23:55:53.802684978 (60h20m24.992983555s) [noise=true , address=104.36.82.5:4839 , piece_id=3GV3PUJ2CLBNYWILAR7SYHJR7WPOKAZEOAUQV33HRZ3D76ASB5LA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:53.802803860 (60h20m24.993102427s) [noise=true , address=104.36.82.8:4972 , piece_id=EH2WQVULBSEWAKV7BHJH4K2QEPETGV5MJECTAVNMYYPHSMW5XJDQ, seg_pos={0 0}, upload=57838]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:53.802829855 (60h20m24.993128421s) [noise=true , address=77.90.31.17:4770 , piece_id=QPI7MIMEZBMM62UGJ4O27UMWKDLWQ5D7C26XZ6WLOWCXVEBEMNBA, seg_pos={0 0}, upload=57838]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:53.802867613 (60h20m24.993166193s) [noise=true , address=104.36.82.11:4113 , piece_id=FABIS2URHW6HOY6HKCZQT2LGILZVAZI4JFB42GBLYVLRCAH6WNDQ, seg_pos={0 0}, upload=57838]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:50876->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:50876->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50876->104.36.82.11:4113: use of closed network connection Jul 26 23:55:53.802905643 (60h20m24.993204225s) [seg_pos={0 0}, upload=57838]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:55:53.927919871 (60h20m25.118218437s) [upload=57838]: Done uploading segments... Jul 26 23:55:53.927945873 (60h20m25.118244453s) [upload=57838]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.128421641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [......XX....X....X.X..X........XX.X....X......X..X..........] duration: 46.183353326s FAIL (2025-07-26 23:55:48 UTC) orbiter warning: context deadline exceeded Duration: 46.18337587s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X..XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X..] duration: 2.964066865s FAIL (2025-07-26 23:55:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.964101886s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..XXX] duration: 30.108465993s FAIL (2025-07-26 23:55:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.108496055s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...X.X.X..X] duration: 1m0.130779179s FAIL (2025-07-26 23:55:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 805]: Upload finished. err: <nil> Jul 26 23:54:28.096334555 (60h18m59.286633135s) [upload=57825]: Starting upload Jul 26 23:54:28.096427911 (60h18m59.286726491s) [upload=57825]: Done starting upload Jul 26 23:54:28.096702226 (60h18m59.287000803s) [upload=57825]: Uploading segments... Jul 26 23:54:28.096900836 (60h18m59.287199431s) [upload=57825]: Got next segment. Inline: false Jul 26 23:54:28.096971617 (60h18m59.287270184s) [upload=57825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:28.096977831 (60h18m59.287276416s) [upload=57825]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:28.299542702 (60h18m59.489841282s) [seg_pos={0 0}, upload=57825]: Begin upload segment... Jul 26 23:54:28.299611901 (60h18m59.489910481s) [seg_pos={0 0}, upload=57825]: Done begin upload segment. Jul 26 23:54:28.299640166 (60h18m59.489938751s) [noise=true , address=104.36.82.56:4248 , piece_id=FMU6WFXEBKQ3XJBFOLFGIR5M2LHJEJC2RY7OQP5C6357CDN4A2DA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300031224 (60h18m59.490329794s) [noise=true , address=104.36.82.53:4130 , piece_id=BCP7RP4BD356BYESTHDFHITTOVWFG3KEPFD5XS2UPETI2ZQ6HITQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300091270 (60h18m59.490389842s) [noise=true , address=79.127.201.215:4346 , piece_id=TKNXFBYJCRUTSBYYPDP7L257VYPLBAXTDBEPAZOADJAOIMUVTR4Q, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300162445 (60h18m59.490461018s) [noise=true , address=77.90.31.21:4896 , piece_id=AF23VOXCUMFHTLOFSL2JBKN4QN5I2JDD4TVFA4POVHI334SVV5IA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300234442 (60h18m59.490533012s) [noise=true , address=79.127.201.214:4321 , piece_id=R6UHK6O3NNQ3YZCMZYNUH6WG4EOYWZZXWE5WFMQEUX5UL4DBE2MA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300303517 (60h18m59.49060209s) [noise=true , address=77.90.31.12:4605 , piece_id=7C5CGCIG3WGX3XGYLZMLSHH5OVWBDKSBN7Q4WDFYYXSWKVTQ6KJA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300366333 (60h18m59.490664905s) [noise=true , address=79.127.205.246:4288 , piece_id=5TAXPEU2X6H65G5U3AOAVVJH37K27V6QWVEGNGZD54F4QS4OWH6A, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300435349 (60h18m59.490733922s) [noise=true , address=104.36.82.14:4245 , piece_id=UT265LPWEB2VH4BXSTN3F5USGLKS35CHV6HR4BQKFSZ3SNADHGWA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300481120 (60h18m59.490779692s) [noise=true , address=77.90.31.13:4659 , piece_id=RJZE7ASOXH3NU5B7SPLND5JM4P4MREVZTNJJFBEH7TQ7WXTEUPHA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300541313 (60h18m59.490839885s) [noise=true , address=104.36.82.47:4524 , piece_id=ALQM7USNHP4GYCA3FVGGFDIE3PUTAEWWMBXZZ25VGIXSQSBWBRCQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300604572 (60h18m59.490903145s) [noise=true , address=79.127.219.38:4006 , piece_id=XU7XJPZQB5VNCLOY3PYAIZMCJCFUABLNJ3VQM7W2XY2WQ465A5OQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300662956 (60h18m59.490961528s) [noise=true , address=104.36.82.54:4179 , piece_id=3DCT2B3O6TG77U3G4PS3MDL2Q4ONN2BMFLOKTQBLEN4ULYJLIF4Q, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300709091 (60h18m59.491007663s) [noise=true , address=104.36.82.11:4132 , piece_id=FJCVE4MO4TSCZYXWZ5GJCUIDWE3OBDTDIVLMFUCER7JVY7GHBGGQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.300765284 (60h18m59.491063856s) [noise=true , address=77.90.31.18:4832 , piece_id=HT5OBOQGBBBD2TXSFUH6XTWYZEI5ZYVPSCABAQPRCWA6SBLGKAWA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.301552953 (60h18m59.491851526s) [noise=true , address=77.90.31.17:4800 , piece_id=3KAJCF6WZ4LMBY6CTFQWVJP6U7DDIBSEO6L6TV4JJ3QHKVA3NPVA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.303412739 (60h18m59.493711311s) [noise=true , address=45.91.116.9:4273 , piece_id=4OQWWOIDPLPBXMHVYHFV4KZRT3IWZCISKQZJYDOITLOXUQRXCYWQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.303947199 (60h18m59.494245772s) [noise=true , address=77.90.31.19:4856 , piece_id=EDHQ5ASTQZN53D7MI6SYV43WYX5YUEBV6MJ6DID6UMLJDUWJDHHQ, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.304457267 (60h18m59.494755833s) [noise=true , address=79.127.205.235:4681 , piece_id=7MVAGJMGMG2OMUPNWY4JE67HR2GXRRY3LZ7BOGICMDCDNFVZP7XA, seg_pos={0 0}, upload=57825]: Uploading piece... Jul 26 23:54:28.880895881 (60h19m0.071194454s) [upload=57825]: Next returned nil segment Jul 26 23:54:28.880917029 (60h19m0.071215598s) [upload=57825]: Waiting for error group managing segments... Jul 26 23:54:28.938759139 (60h19m0.12905772s) [noise=true , address=79.127.201.214:4321 , piece_id=R6UHK6O3NNQ3YZCMZYNUH6WG4EOYWZZXWE5WFMQEUX5UL4DBE2MA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:54:29.022625594 (60h19m0.21292418s) [noise=true , address=45.91.116.9:4273 , piece_id=4OQWWOIDPLPBXMHVYHFV4KZRT3IWZCISKQZJYDOITLOXUQRXCYWQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.060699428 (60h19m0.250998001s) [noise=true , address=79.127.219.38:4006 , piece_id=XU7XJPZQB5VNCLOY3PYAIZMCJCFUABLNJ3VQM7W2XY2WQ465A5OQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.535800102 (60h19m0.726098682s) [noise=true , address=79.127.205.235:4681 , piece_id=7MVAGJMGMG2OMUPNWY4JE67HR2GXRRY3LZ7BOGICMDCDNFVZP7XA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:29.597980265 (60h19m0.78827883s) [noise=true , address=79.127.205.246:4288 , piece_id=5TAXPEU2X6H65G5U3AOAVVJH37K27V6QWVEGNGZD54F4QS4OWH6A, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:40.523474364 (60h19m11.713772929s) [noise=true , address=77.90.31.19:4856 , piece_id=EDHQ5ASTQZN53D7MI6SYV43WYX5YUEBV6MJ6DID6UMLJDUWJDHHQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:47.746672172 (60h19m18.936970744s) [noise=true , address=79.127.201.215:4346 , piece_id=TKNXFBYJCRUTSBYYPDP7L257VYPLBAXTDBEPAZOADJAOIMUVTR4Q, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:48.228273050 (60h19m19.418571622s) [noise=true , address=104.36.82.53:4130 , piece_id=BCP7RP4BD356BYESTHDFHITTOVWFG3KEPFD5XS2UPETI2ZQ6HITQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:51.426042821 (60h19m22.616341394s) [noise=true , address=77.90.31.12:4605 , piece_id=7C5CGCIG3WGX3XGYLZMLSHH5OVWBDKSBN7Q4WDFYYXSWKVTQ6KJA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:54.360685494 (60h19m25.550984067s) [noise=true , address=104.36.82.56:4248 , piece_id=FMU6WFXEBKQ3XJBFOLFGIR5M2LHJEJC2RY7OQP5C6357CDN4A2DA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:55.549356195 (60h19m26.739654764s) [noise=true , address=77.90.31.13:4659 , piece_id=RJZE7ASOXH3NU5B7SPLND5JM4P4MREVZTNJJFBEH7TQ7WXTEUPHA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:54:58.159243977 (60h19m29.349542555s) [noise=true , address=104.36.82.54:4179 , piece_id=3DCT2B3O6TG77U3G4PS3MDL2Q4ONN2BMFLOKTQBLEN4ULYJLIF4Q, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:06.621983856 (60h19m37.812282426s) [noise=true , address=104.36.82.47:4524 , piece_id=ALQM7USNHP4GYCA3FVGGFDIE3PUTAEWWMBXZZ25VGIXSQSBWBRCQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:24.918176111 (60h19m56.108474678s) [noise=true , address=104.36.82.11:4132 , piece_id=FJCVE4MO4TSCZYXWZ5GJCUIDWE3OBDTDIVLMFUCER7JVY7GHBGGQ, seg_pos={0 0}, upload=57825]: Done uploading piece. err: <nil> Jul 26 23:55:27.822461227 (60h19m59.012759806s) [noise=true , address=77.90.31.21:4896 , piece_id=AF23VOXCUMFHTLOFSL2JBKN4QN5I2JDD4TVFA4POVHI334SVV5IA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:27.822573319 (60h19m59.012871899s) [noise=true , address=104.36.82.14:4245 , piece_id=UT265LPWEB2VH4BXSTN3F5USGLKS35CHV6HR4BQKFSZ3SNADHGWA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:55:27.822629848 (60h19m59.012928429s) [noise=true , address=77.90.31.17:4800 , piece_id=3KAJCF6WZ4LMBY6CTFQWVJP6U7DDIBSEO6L6TV4JJ3QHKVA3NPVA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: rpc: dial tcp 77.90.31.17:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:27.822650434 (60h19m59.012949014s) [noise=true , address=77.90.31.18:4832 , piece_id=HT5OBOQGBBBD2TXSFUH6XTWYZEI5ZYVPSCABAQPRCWA6SBLGKAWA, seg_pos={0 0}, upload=57825]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: write tcp 139.59.144.22:50530->77.90.31.18:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:55:27.822698989 (60h19m59.012997568s) [seg_pos={0 0}, upload=57825]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:55:27.950123107 (60h19m59.140421672s) [upload=57825]: Done uploading segments... Jul 26 23:55:27.950146794 (60h19m59.140445372s) [upload=57825]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.13080245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 751.570391ms FAIL (2025-07-26 23:55:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 751.605687ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X..XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X.] duration: 4.780792029s FAIL (2025-07-26 23:55:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.780817784s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.XX.] duration: 20.212656707s FAIL (2025-07-26 23:55:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== . successful: 11 optimal: 13 errs: <nil> Jul 26 23:53:58.180562800 (60h18m29.370861367s) [upload=57807]: Done uploading segments... Jul 26 23:53:58.180590687 (60h18m29.370889252s) [upload=57807]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:54:26.950854225 (60h18m58.141152815s) [upload=57824]: Starting upload Jul 26 23:54:26.950934307 (60h18m58.141232896s) [upload=57824]: Done starting upload Jul 26 23:54:26.951187097 (60h18m58.141485664s) [upload=57824]: Uploading segments... Jul 26 23:54:26.951226134 (60h18m58.141524721s) [upload=57824]: Got next segment. Inline: false Jul 26 23:54:26.951257266 (60h18m58.141555831s) [upload=57824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:26.951261970 (60h18m58.141560535s) [upload=57824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:27.060384169 (60h18m58.250682754s) [seg_pos={0 0}, upload=57824]: Begin upload segment... Jul 26 23:54:27.060467317 (60h18m58.250765915s) [noise=true , address=104.36.82.4:4803 , piece_id=FWCXWBKSD6I3KPNQFHSYM5QNXE2STCMS6Y2OQVSNQLDDSVYAORBQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.060833158 (60h18m58.251131739s) [seg_pos={0 0}, upload=57824]: Done begin upload segment. Jul 26 23:54:27.060855485 (60h18m58.251154067s) [noise=true , address=104.36.82.15:4259 , piece_id=XUDGRCQNXEVZPTYFUIHLG5DEXGKUS4XA6ICM5KCKMSIOYQIANSYA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.061235187 (60h18m58.25153377s) [noise=true , address=45.91.116.10:4081 , piece_id=YWCVO2HMMUGRQLFVLZQBIKKXUAXJPTNGDFAQVJCTWF4DHJMRYX7Q, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.062010535 (60h18m58.252309127s) [noise=true , address=77.90.31.15:4699 , piece_id=ITIYPTBQEGCUADYDNEEOHB7QNJHJ7HREJHEPMPJJZHV2G5E3LLIA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.063172508 (60h18m58.253471097s) [noise=true , address=45.91.116.4:4937 , piece_id=MRKYHA4IJJ6NTJY5C3DVPXURMQQRGZ3VE7RA4PE3RX3PD6LNDPMQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.064262736 (60h18m58.254561323s) [noise=true , address=45.91.116.13:4153 , piece_id=Z254E2QLQHJAKIMO4YXE7DWH574YJL65YHBEDYTMTAP5WLWKUMYA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.064478362 (60h18m58.254776946s) [noise=true , address=77.90.31.19:4841 , piece_id=CZQXKGHEN6J7T7YKHPNUFXDDDUHDUEXD3572Y2B5YT6IMHDMP4RQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.065144780 (60h18m58.255443368s) [noise=true , address=45.91.116.12:4226 , piece_id=7J2EA7COBHFEIIOF7DMGEH7B5MHFZ3VW5J5QFNYZVRYMVGRNQ7BQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.065799412 (60h18m58.256097992s) [noise=true , address=77.90.31.16:4733 , piece_id=OMDZZK32GDRMLZ7R2IYEGASGYWXVG3EQEB7EF7DOCUJKUGNDO52Q, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.066222021 (60h18m58.256520607s) [noise=true , address=45.91.116.6:4973 , piece_id=LSQREHFWLXXPM5UJZJTQVAEYIPKRDB3RWOT2XF3XTLCL6VG6DM3A, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.066568029 (60h18m58.256866608s) [noise=true , address=104.36.82.5:4840 , piece_id=R3VNEG2CHDYYTUCUTFSZRV4T5DZTSOMJP2FISWB76ES7EY5LDT5Q, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.067549883 (60h18m58.257848465s) [noise=true , address=45.91.116.7:4009 , piece_id=RG5BN4YSFC2LIRF66AXKP3XUCW24Q4NFEHQEN7P5MDBCQLCZLDIQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.068126220 (60h18m58.258424807s) [noise=true , address=45.91.116.9:4263 , piece_id=QPULHJQD6KOWJJ3RMWL6WLSPB2FMX5YDMR5DBTGGVMJTMUA5FOGA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.068962756 (60h18m58.259261345s) [noise=true , address=104.36.82.6:4874 , piece_id=YJAVYS7MCW2GTYYOSHLY7Y2PY7PTVG2U56EQWQHQDZ2DHYSDVGCA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.070026392 (60h18m58.260324978s) [noise=true , address=45.91.116.8:4048 , piece_id=JGXNKUDL3II65NXDRVV5K5GXHCAAIF7NDLGNAGCZWQU25JEMNZWA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.070635756 (60h18m58.260934368s) [noise=true , address=77.90.31.13:4628 , piece_id=DDQBIXMS7MPT65APDKJEJYQHZ6RHRYMCAWTJOI3DEZX7FBA2FEKQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.679571040 (60h18m58.869869623s) [upload=57824]: Next returned nil segment Jul 26 23:54:27.679593294 (60h18m58.869891884s) [upload=57824]: Waiting for error group managing segments... Jul 26 23:54:27.807547848 (60h18m58.997846417s) [noise=true , address=77.90.31.15:4699 , piece_id=ITIYPTBQEGCUADYDNEEOHB7QNJHJ7HREJHEPMPJJZHV2G5E3LLIA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.811129797 (60h18m59.00142837s) [noise=true , address=45.91.116.13:4153 , piece_id=Z254E2QLQHJAKIMO4YXE7DWH574YJL65YHBEDYTMTAP5WLWKUMYA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.813328193 (60h18m59.00362676s) [noise=true , address=77.90.31.16:4733 , piece_id=OMDZZK32GDRMLZ7R2IYEGASGYWXVG3EQEB7EF7DOCUJKUGNDO52Q, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.817636152 (60h18m59.007934723s) [noise=true , address=45.91.116.10:4081 , piece_id=YWCVO2HMMUGRQLFVLZQBIKKXUAXJPTNGDFAQVJCTWF4DHJMRYX7Q, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.821472865 (60h18m59.011771438s) [noise=true , address=45.91.116.12:4226 , piece_id=7J2EA7COBHFEIIOF7DMGEH7B5MHFZ3VW5J5QFNYZVRYMVGRNQ7BQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.838068845 (60h18m59.028367417s) [noise=true , address=45.91.116.7:4009 , piece_id=RG5BN4YSFC2LIRF66AXKP3XUCW24Q4NFEHQEN7P5MDBCQLCZLDIQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.842495970 (60h18m59.032794549s) [noise=true , address=45.91.116.6:4973 , piece_id=LSQREHFWLXXPM5UJZJTQVAEYIPKRDB3RWOT2XF3XTLCL6VG6DM3A, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.844128639 (60h18m59.034427211s) [noise=true , address=45.91.116.8:4048 , piece_id=JGXNKUDL3II65NXDRVV5K5GXHCAAIF7NDLGNAGCZWQU25JEMNZWA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.901574395 (60h18m59.091872975s) [noise=true , address=45.91.116.4:4937 , piece_id=MRKYHA4IJJ6NTJY5C3DVPXURMQQRGZ3VE7RA4PE3RX3PD6LNDPMQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.982330588 (60h18m59.172629164s) [noise=true , address=45.91.116.9:4263 , piece_id=QPULHJQD6KOWJJ3RMWL6WLSPB2FMX5YDMR5DBTGGVMJTMUA5FOGA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:29.233862392 (60h19m0.424160962s) [noise=true , address=77.90.31.19:4841 , piece_id=CZQXKGHEN6J7T7YKHPNUFXDDDUHDUEXD3572Y2B5YT6IMHDMP4RQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:29.298662861 (60h19m0.488961447s) [noise=true , address=77.90.31.13:4628 , piece_id=DDQBIXMS7MPT65APDKJEJYQHZ6RHRYMCAWTJOI3DEZX7FBA2FEKQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:56.678760508 (60h19m27.869059078s) [noise=true , address=104.36.82.6:4874 , piece_id=YJAVYS7MCW2GTYYOSHLY7Y2PY7PTVG2U56EQWQHQDZ2DHYSDVGCA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:55448->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:55448->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55448->104.36.82.6:4874: use of closed network connection Jul 26 23:54:56.678811142 (60h19m27.869109712s) [noise=true , address=104.36.82.4:4803 , piece_id=FWCXWBKSD6I3KPNQFHSYM5QNXE2STCMS6Y2OQVSNQLDDSVYAORBQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:51692->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:51692->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51692->104.36.82.4:4803: use of closed network connection Jul 26 23:54:56.678844819 (60h19m27.869143387s) [noise=true , address=104.36.82.15:4259 , piece_id=XUDGRCQNXEVZPTYFUIHLG5DEXGKUS4XA6ICM5KCKMSIOYQIANSYA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 139.59.144.22:34348->104.36.82.15:4259: use of closed network connection; write tcp 139.59.144.22:34348->104.36.82.15:4259: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34348->104.36.82.15:4259: use of closed network connection Jul 26 23:54:56.678883645 (60h19m27.869182214s) [noise=true , address=104.36.82.5:4840 , piece_id=R3VNEG2CHDYYTUCUTFSZRV4T5DZTSOMJP2FISWB76ES7EY5LDT5Q, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 139.59.144.22:50930->104.36.82.5:4840: use of closed network connection; write tcp 139.59.144.22:50930->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50930->104.36.82.5:4840: use of closed network connection Jul 26 23:54:56.678902476 (60h19m27.869201062s) [seg_pos={0 0}, upload=57824]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:54:56.787856845 (60h19m27.978155417s) [upload=57824]: Done uploading segments... Jul 26 23:54:56.787892955 (60h19m27.978191524s) [upload=57824]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 20.21271275s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.......XX....X....X.X..X........XX.X....X......X..X.........] duration: 40.583470121s FAIL (2025-07-26 23:55:02 UTC) orbiter warning: context deadline exceeded Duration: 40.583511366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.XX] duration: 30.112530346s FAIL (2025-07-26 23:54:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== . successful: 11 optimal: 13 errs: <nil> Jul 26 23:53:58.180562800 (60h18m29.370861367s) [upload=57807]: Done uploading segments... Jul 26 23:53:58.180590687 (60h18m29.370889252s) [upload=57807]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:54:26.950854225 (60h18m58.141152815s) [upload=57824]: Starting upload Jul 26 23:54:26.950934307 (60h18m58.141232896s) [upload=57824]: Done starting upload Jul 26 23:54:26.951187097 (60h18m58.141485664s) [upload=57824]: Uploading segments... Jul 26 23:54:26.951226134 (60h18m58.141524721s) [upload=57824]: Got next segment. Inline: false Jul 26 23:54:26.951257266 (60h18m58.141555831s) [upload=57824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:54:26.951261970 (60h18m58.141560535s) [upload=57824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:54:27.060384169 (60h18m58.250682754s) [seg_pos={0 0}, upload=57824]: Begin upload segment... Jul 26 23:54:27.060467317 (60h18m58.250765915s) [noise=true , address=104.36.82.4:4803 , piece_id=FWCXWBKSD6I3KPNQFHSYM5QNXE2STCMS6Y2OQVSNQLDDSVYAORBQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.060833158 (60h18m58.251131739s) [seg_pos={0 0}, upload=57824]: Done begin upload segment. Jul 26 23:54:27.060855485 (60h18m58.251154067s) [noise=true , address=104.36.82.15:4259 , piece_id=XUDGRCQNXEVZPTYFUIHLG5DEXGKUS4XA6ICM5KCKMSIOYQIANSYA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.061235187 (60h18m58.25153377s) [noise=true , address=45.91.116.10:4081 , piece_id=YWCVO2HMMUGRQLFVLZQBIKKXUAXJPTNGDFAQVJCTWF4DHJMRYX7Q, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.062010535 (60h18m58.252309127s) [noise=true , address=77.90.31.15:4699 , piece_id=ITIYPTBQEGCUADYDNEEOHB7QNJHJ7HREJHEPMPJJZHV2G5E3LLIA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.063172508 (60h18m58.253471097s) [noise=true , address=45.91.116.4:4937 , piece_id=MRKYHA4IJJ6NTJY5C3DVPXURMQQRGZ3VE7RA4PE3RX3PD6LNDPMQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.064262736 (60h18m58.254561323s) [noise=true , address=45.91.116.13:4153 , piece_id=Z254E2QLQHJAKIMO4YXE7DWH574YJL65YHBEDYTMTAP5WLWKUMYA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.064478362 (60h18m58.254776946s) [noise=true , address=77.90.31.19:4841 , piece_id=CZQXKGHEN6J7T7YKHPNUFXDDDUHDUEXD3572Y2B5YT6IMHDMP4RQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.065144780 (60h18m58.255443368s) [noise=true , address=45.91.116.12:4226 , piece_id=7J2EA7COBHFEIIOF7DMGEH7B5MHFZ3VW5J5QFNYZVRYMVGRNQ7BQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.065799412 (60h18m58.256097992s) [noise=true , address=77.90.31.16:4733 , piece_id=OMDZZK32GDRMLZ7R2IYEGASGYWXVG3EQEB7EF7DOCUJKUGNDO52Q, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.066222021 (60h18m58.256520607s) [noise=true , address=45.91.116.6:4973 , piece_id=LSQREHFWLXXPM5UJZJTQVAEYIPKRDB3RWOT2XF3XTLCL6VG6DM3A, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.066568029 (60h18m58.256866608s) [noise=true , address=104.36.82.5:4840 , piece_id=R3VNEG2CHDYYTUCUTFSZRV4T5DZTSOMJP2FISWB76ES7EY5LDT5Q, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.067549883 (60h18m58.257848465s) [noise=true , address=45.91.116.7:4009 , piece_id=RG5BN4YSFC2LIRF66AXKP3XUCW24Q4NFEHQEN7P5MDBCQLCZLDIQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.068126220 (60h18m58.258424807s) [noise=true , address=45.91.116.9:4263 , piece_id=QPULHJQD6KOWJJ3RMWL6WLSPB2FMX5YDMR5DBTGGVMJTMUA5FOGA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.068962756 (60h18m58.259261345s) [noise=true , address=104.36.82.6:4874 , piece_id=YJAVYS7MCW2GTYYOSHLY7Y2PY7PTVG2U56EQWQHQDZ2DHYSDVGCA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.070026392 (60h18m58.260324978s) [noise=true , address=45.91.116.8:4048 , piece_id=JGXNKUDL3II65NXDRVV5K5GXHCAAIF7NDLGNAGCZWQU25JEMNZWA, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.070635756 (60h18m58.260934368s) [noise=true , address=77.90.31.13:4628 , piece_id=DDQBIXMS7MPT65APDKJEJYQHZ6RHRYMCAWTJOI3DEZX7FBA2FEKQ, seg_pos={0 0}, upload=57824]: Uploading piece... Jul 26 23:54:27.679571040 (60h18m58.869869623s) [upload=57824]: Next returned nil segment Jul 26 23:54:27.679593294 (60h18m58.869891884s) [upload=57824]: Waiting for error group managing segments... Jul 26 23:54:27.807547848 (60h18m58.997846417s) [noise=true , address=77.90.31.15:4699 , piece_id=ITIYPTBQEGCUADYDNEEOHB7QNJHJ7HREJHEPMPJJZHV2G5E3LLIA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.811129797 (60h18m59.00142837s) [noise=true , address=45.91.116.13:4153 , piece_id=Z254E2QLQHJAKIMO4YXE7DWH574YJL65YHBEDYTMTAP5WLWKUMYA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.813328193 (60h18m59.00362676s) [noise=true , address=77.90.31.16:4733 , piece_id=OMDZZK32GDRMLZ7R2IYEGASGYWXVG3EQEB7EF7DOCUJKUGNDO52Q, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.817636152 (60h18m59.007934723s) [noise=true , address=45.91.116.10:4081 , piece_id=YWCVO2HMMUGRQLFVLZQBIKKXUAXJPTNGDFAQVJCTWF4DHJMRYX7Q, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.821472865 (60h18m59.011771438s) [noise=true , address=45.91.116.12:4226 , piece_id=7J2EA7COBHFEIIOF7DMGEH7B5MHFZ3VW5J5QFNYZVRYMVGRNQ7BQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.838068845 (60h18m59.028367417s) [noise=true , address=45.91.116.7:4009 , piece_id=RG5BN4YSFC2LIRF66AXKP3XUCW24Q4NFEHQEN7P5MDBCQLCZLDIQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.842495970 (60h18m59.032794549s) [noise=true , address=45.91.116.6:4973 , piece_id=LSQREHFWLXXPM5UJZJTQVAEYIPKRDB3RWOT2XF3XTLCL6VG6DM3A, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.844128639 (60h18m59.034427211s) [noise=true , address=45.91.116.8:4048 , piece_id=JGXNKUDL3II65NXDRVV5K5GXHCAAIF7NDLGNAGCZWQU25JEMNZWA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.901574395 (60h18m59.091872975s) [noise=true , address=45.91.116.4:4937 , piece_id=MRKYHA4IJJ6NTJY5C3DVPXURMQQRGZ3VE7RA4PE3RX3PD6LNDPMQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:27.982330588 (60h18m59.172629164s) [noise=true , address=45.91.116.9:4263 , piece_id=QPULHJQD6KOWJJ3RMWL6WLSPB2FMX5YDMR5DBTGGVMJTMUA5FOGA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:29.233862392 (60h19m0.424160962s) [noise=true , address=77.90.31.19:4841 , piece_id=CZQXKGHEN6J7T7YKHPNUFXDDDUHDUEXD3572Y2B5YT6IMHDMP4RQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:29.298662861 (60h19m0.488961447s) [noise=true , address=77.90.31.13:4628 , piece_id=DDQBIXMS7MPT65APDKJEJYQHZ6RHRYMCAWTJOI3DEZX7FBA2FEKQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: <nil> Jul 26 23:54:56.678760508 (60h19m27.869059078s) [noise=true , address=104.36.82.6:4874 , piece_id=YJAVYS7MCW2GTYYOSHLY7Y2PY7PTVG2U56EQWQHQDZ2DHYSDVGCA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:55448->104.36.82.6:4874: use of closed network connection; write tcp 139.59.144.22:55448->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55448->104.36.82.6:4874: use of closed network connection Jul 26 23:54:56.678811142 (60h19m27.869109712s) [noise=true , address=104.36.82.4:4803 , piece_id=FWCXWBKSD6I3KPNQFHSYM5QNXE2STCMS6Y2OQVSNQLDDSVYAORBQ, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:51692->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:51692->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51692->104.36.82.4:4803: use of closed network connection Jul 26 23:54:56.678844819 (60h19m27.869143387s) [noise=true , address=104.36.82.15:4259 , piece_id=XUDGRCQNXEVZPTYFUIHLG5DEXGKUS4XA6ICM5KCKMSIOYQIANSYA, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 139.59.144.22:34348->104.36.82.15:4259: use of closed network connection; write tcp 139.59.144.22:34348->104.36.82.15:4259: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34348->104.36.82.15:4259: use of closed network connection Jul 26 23:54:56.678883645 (60h19m27.869182214s) [noise=true , address=104.36.82.5:4840 , piece_id=R3VNEG2CHDYYTUCUTFSZRV4T5DZTSOMJP2FISWB76ES7EY5LDT5Q, seg_pos={0 0}, upload=57824]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 139.59.144.22:50930->104.36.82.5:4840: use of closed network connection; write tcp 139.59.144.22:50930->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50930->104.36.82.5:4840: use of closed network connection Jul 26 23:54:56.678902476 (60h19m27.869201062s) [seg_pos={0 0}, upload=57824]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:54:56.787856845 (60h19m27.978155417s) [upload=57824]: Done uploading segments... Jul 26 23:54:56.787892955 (60h19m27.978191524s) [upload=57824]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.112558609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..XX] duration: 30.105909737s FAIL (2025-07-26 23:54:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.105950568s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X..XXXXX....XX.XX..X.......X......X..XXX....XXX...X....X] duration: 30.001291035s FAIL (2025-07-26 23:54:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001320943s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..X] duration: 30.121395726s FAIL (2025-07-26 23:54:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.121423531s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...X.X.X..] duration: 54.047875514s FAIL (2025-07-26 23:54:16 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=57760]: Upload finished. err: <nil> Jul 26 23:51:21.120844821 (60h15m52.311143392s) [upload=57777]: Starting upload Jul 26 23:51:21.120914819 (60h15m52.311213398s) [upload=57777]: Done starting upload Jul 26 23:51:21.121114266 (60h15m52.311412834s) [upload=57777]: Uploading segments... Jul 26 23:51:21.121315062 (60h15m52.311613645s) [upload=57777]: Got next segment. Inline: false Jul 26 23:51:21.121373173 (60h15m52.311671755s) [upload=57777]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:51:21.121393948 (60h15m52.31169253s) [upload=57777]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:51:21.318613089 (60h15m52.508911668s) [seg_pos={0 0}, upload=57777]: Begin upload segment... Jul 26 23:51:21.318678238 (60h15m52.508976817s) [seg_pos={0 0}, upload=57777]: Done begin upload segment. Jul 26 23:51:21.318709310 (60h15m52.509007893s) [noise=true , address=77.90.31.16:4737 , piece_id=74J35KLDFKHQEDFC5P5UWXF2CQNMGB5TETBQ2DY7OGEPERC5FJAQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.318772604 (60h15m52.509074513s) [noise=true , address=79.127.201.215:4357 , piece_id=H7WBF5RSI5VC6LHJ6KRPFEDP46V2WBY32QDIOHA6OW4SAEGYAR2Q, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.319158495 (60h15m52.509457077s) [noise=true , address=77.90.31.18:4823 , piece_id=IFSA66FBFC6X6GXBPEDSPFMCOQJR4X7CYAQXEDVDE5BYLTTEPK2A, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.319526702 (60h15m52.509825281s) [noise=true , address=104.36.82.14:4228 , piece_id=6F5FHJHJKWLDZZ7BJK5PTU4OAWLVVGKDYYSCRRA46P5CPD2P6VWQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.319890417 (60h15m52.510188998s) [noise=true , address=104.36.82.46:4516 , piece_id=AZIVYGF7HGC3U2DYRF7GIXYBAY3JW352FHL6IME6F6IYOA2QQTZA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.320323126 (60h15m52.510621718s) [noise=true , address=77.90.31.13:4648 , piece_id=U6TE7ZOCAW3CUVFRS3XW3FBLEC6DR6AY2SSFAEW6OS5CX3GAJV6Q, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.320812188 (60h15m52.511110776s) [noise=true , address=77.90.31.19:4867 , piece_id=FITJA6MPO6VWH2GJ47VAWJIJAZ3ZV6GK6WPLWOXE5M42GRMDWJUQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.321354390 (60h15m52.511652976s) [noise=true , address=104.36.82.3:4778 , piece_id=NLCDIH7LKKBHVB6PM2ZPGD7AQAGHQ5GF23D3CBMN3SWJDB3JMMPQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.321561681 (60h15m52.511860348s) [noise=true , address=104.36.82.54:4192 , piece_id=B7TUN6TFD6C54XAC6GXXLQJSBI3727647TS2K7ZGNW36U2J2HUHQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322119097 (60h15m52.512417683s) [noise=true , address=64.49.13.140:4508 , piece_id=35SDQJFL3BDYX7DTIF6YIP3JKUG6XCREWF5ETOZCPM3R25OM55UA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322487621 (60h15m52.512786208s) [noise=true , address=79.127.201.215:4353 , piece_id=XURH3ZG2VEEUOGYTV6IOLVNAD5G3SFFAYYJELKL6L4FLD4U6YWUA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322553663 (60h15m52.512852252s) [noise=true , address=79.127.219.38:4018 , piece_id=OMXH75H2YDMVFXSISNUI5EFQMWP6Z7IXV6E56BYIL5PPNCNWCXQQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322855840 (60h15m52.513154423s) [noise=true , address=79.127.205.235:4685 , piece_id=O3LIB5E5AQ4GNEDUHLKPYWEXBCNKYS75EMT7PXCL26WZZSXTLXSA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.323201594 (60h15m52.513500174s) [noise=true , address=156.146.43.225:4434 , piece_id=XC66CSU3RHGMFIYYANZ2INZFOJ3VT5WO2KQWNIUKRCJPWENAKYAQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.323634016 (60h15m52.513932598s) [noise=true , address=104.36.82.53:4134 , piece_id=XRWBV5FUK5DZ24YLIZQWDTFWSC3XSM7YDTKYF3Y63N2DR6PYHEYA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.323866203 (60h15m52.514164791s) [noise=true , address=77.90.31.15:4714 , piece_id=RUZRS4S2WJENYK2NZWHRFXHFCRJCQ677IXXSQJUS4YO56R7GK2XA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.324366738 (60h15m52.514665321s) [noise=true , address=79.127.201.215:4360 , piece_id=XE4Q2S5TAXK7C5HYZMBZLJR66AEGKQSPYI3ZTDUEHTUOIX3GECAA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.324716629 (60h15m52.515015218s) [noise=true , address=79.127.205.246:4284 , piece_id=QPRZOSHJFD272ITYE2KIIXNMHUCE6VCE6ARRTYVTZZVUU462MT7A, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.880171655 (60h15m53.070470234s) [upload=57777]: Next returned nil segment Jul 26 23:51:21.880192137 (60h15m53.070490717s) [upload=57777]: Waiting for error group managing segments... Jul 26 23:51:22.070866647 (60h15m53.261165222s) [noise=true , address=77.90.31.16:4737 , piece_id=74J35KLDFKHQEDFC5P5UWXF2CQNMGB5TETBQ2DY7OGEPERC5FJAQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.131675699 (60h15m53.321974281s) [noise=true , address=79.127.219.38:4018 , piece_id=OMXH75H2YDMVFXSISNUI5EFQMWP6Z7IXV6E56BYIL5PPNCNWCXQQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.337500020 (60h15m53.527798604s) [noise=true , address=79.127.201.215:4357 , piece_id=H7WBF5RSI5VC6LHJ6KRPFEDP46V2WBY32QDIOHA6OW4SAEGYAR2Q, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.655708148 (60h15m53.846006715s) [noise=true , address=77.90.31.15:4714 , piece_id=RUZRS4S2WJENYK2NZWHRFXHFCRJCQ677IXXSQJUS4YO56R7GK2XA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: storage node overloaded, request limit: 2000 Jul 26 23:51:22.669727155 (60h15m53.860025739s) [noise=true , address=79.127.205.235:4685 , piece_id=O3LIB5E5AQ4GNEDUHLKPYWEXBCNKYS75EMT7PXCL26WZZSXTLXSA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.692515988 (60h15m53.882814634s) [noise=true , address=79.127.205.246:4284 , piece_id=QPRZOSHJFD272ITYE2KIIXNMHUCE6VCE6ARRTYVTZZVUU462MT7A, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.927479731 (60h15m54.117778296s) [noise=true , address=64.49.13.140:4508 , piece_id=35SDQJFL3BDYX7DTIF6YIP3JKUG6XCREWF5ETOZCPM3R25OM55UA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:23.572960259 (60h15m54.763258828s) [noise=true , address=156.146.43.225:4434 , piece_id=XC66CSU3RHGMFIYYANZ2INZFOJ3VT5WO2KQWNIUKRCJPWENAKYAQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:35.403823526 (60h16m6.594122095s) [noise=true , address=77.90.31.19:4867 , piece_id=FITJA6MPO6VWH2GJ47VAWJIJAZ3ZV6GK6WPLWOXE5M42GRMDWJUQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:37.584576677 (60h16m8.77487526s) [noise=true , address=79.127.201.215:4360 , piece_id=XE4Q2S5TAXK7C5HYZMBZLJR66AEGKQSPYI3ZTDUEHTUOIX3GECAA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:37.867812853 (60h16m9.058111422s) [noise=true , address=79.127.201.215:4353 , piece_id=XURH3ZG2VEEUOGYTV6IOLVNAD5G3SFFAYYJELKL6L4FLD4U6YWUA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:48.553560938 (60h16m19.743859517s) [noise=true , address=104.36.82.3:4778 , piece_id=NLCDIH7LKKBHVB6PM2ZPGD7AQAGHQ5GF23D3CBMN3SWJDB3JMMPQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:49.779710791 (60h16m20.970009371s) [noise=true , address=104.36.82.54:4192 , piece_id=B7TUN6TFD6C54XAC6GXXLQJSBI3727647TS2K7ZGNW36U2J2HUHQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:52:09.765619693 (60h16m40.955918273s) [noise=true , address=104.36.82.46:4516 , piece_id=AZIVYGF7HGC3U2DYRF7GIXYBAY3JW352FHL6IME6F6IYOA2QQTZA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:52:11.635776353 (60h16m42.826074922s) [noise=true , address=104.36.82.53:4134 , piece_id=XRWBV5FUK5DZ24YLIZQWDTFWSC3XSM7YDTKYF3Y63N2DR6PYHEYA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:52:11.635805183 (60h16m42.826103762s) [seg_pos={0 0}, upload=57777]: Segment reached optimal threshold of 14 pieces. Jul 26 23:52:11.636019615 (60h16m42.826318195s) [noise=true , address=77.90.31.13:4648 , piece_id=U6TE7ZOCAW3CUVFRS3XW3FBLEC6DR6AY2SSFAEW6OS5CX3GAJV6Q, seg_pos={0 0}, upload=57777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh): protocol: expected piece hash; context canceled; EOF Jul 26 23:52:11.636039654 (60h16m42.826338233s) [noise=true , address=104.36.82.14:4228 , piece_id=6F5FHJHJKWLDZZ7BJK5PTU4OAWLVVGKDYYSCRRA46P5CPD2P6VWQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF): protocol: expected piece hash; context canceled; EOF Jul 26 23:52:11.636194965 (60h16m42.826493545s) [noise=true , address=77.90.31.18:4823 , piece_id=IFSA66FBFC6X6GXBPEDSPFMCOQJR4X7CYAQXEDVDE5BYLTTEPK2A, seg_pos={0 0}, upload=57777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: context canceled; context canceled Jul 26 23:52:11.636214607 (60h16m42.826513174s) [seg_pos={0 0}, upload=57777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 23:52:11.636289661 (60h16m42.826588241s) [upload=57777]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:52:11.636294984 (60h16m42.826593563s) [upload=57777]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:52:11.791908693 (60h16m42.982207263s) [upload=57777]: Done uploading segments... Jul 26 23:52:11.791928887 (60h16m42.982227457s) [upload=57777]: Upload finished. err: <nil> Duration: 54.047901651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.X] duration: 30.105603297s FAIL (2025-07-26 23:53:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 6 (60h17m59.538752985s) [upload=57807]: Done starting upload Jul 26 23:53:28.348604281 (60h17m59.538902852s) [upload=57807]: Uploading segments... Jul 26 23:53:28.348954550 (60h17m59.539253132s) [upload=57807]: Got next segment. Inline: false Jul 26 23:53:28.348998434 (60h17m59.539297003s) [upload=57807]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:53:28.349125851 (60h17m59.539424431s) [upload=57807]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:53:28.456298125 (60h17m59.646596707s) [seg_pos={0 0}, upload=57807]: Begin upload segment... Jul 26 23:53:28.456392249 (60h17m59.646690831s) [seg_pos={0 0}, upload=57807]: Done begin upload segment. Jul 26 23:53:28.456440284 (60h17m59.646738873s) [noise=true , address=104.36.82.3:4767 , piece_id=OCULMRJRZCGYVA3BOHJ2SE754TI3BD4IAPIRC3F5BOLQRITCSMBQ, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.456455383 (60h17m59.646753965s) [noise=true , address=104.36.82.3:4765 , piece_id=NJPLIIP2OI6TAYPXNBL5ZPLMTNQTTPQHVOUVNRDA77JVNTDMFXRA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.456854684 (60h17m59.647153266s) [noise=true , address=104.36.82.8:4971 , piece_id=3WEYFTBCZAOXS2EUJXGJCF2U6GJLNG6EHB77FNPTC5RTC2I3SFVA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.456955278 (60h17m59.647253865s) [noise=true , address=45.91.116.9:4261 , piece_id=LVPGW7GYIEPKASCMVDJ2R4PPKYEZEHIPS253HTHY76XXRJW3SY7Q, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.457284789 (60h17m59.647583371s) [noise=true , address=77.90.31.11:4555 , piece_id=KQ6B65YJMNGGH4BFEVXJL4O6ETZGUG37INGHXAYCEFW7XDK72FFQ, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.457543919 (60h17m59.64784251s) [noise=true , address=45.91.116.12:4225 , piece_id=JA2M4UJRTF2HLVYK66NBLM6TN5LZO42OMOFJ4WIWELWG6A3LD6QA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.457585573 (60h17m59.647884155s) [noise=true , address=45.91.116.5:4298 , piece_id=Q4LX4S6G5JMMTIFO7IRF3RO6CFDGIM2GYXELYSEK327QH6XMI3FA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.457872447 (60h17m59.64817104s) [noise=true , address=45.91.116.8:4048 , piece_id=GJ5YGAXPFFB427XM6XRWVLR4JKVYD5OJ3EBNYBKBXQB5MMRRI2VA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.457962470 (60h17m59.648261049s) [noise=true , address=77.90.31.10:4520 , piece_id=XTGECWA4U3HUJZOZKS737KV64T3E6ZNFVQFC24IU7Q6TDLZXMR6Q, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.458345574 (60h17m59.648644156s) [noise=true , address=45.91.116.5:4297 , piece_id=HWEF7KAKGUO5NMJEFUQSZ7M7Y4JYMUVYQMSIFEPIBJT5M3QAEIDQ, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.458760566 (60h17m59.649059159s) [noise=true , address=45.91.116.10:4083 , piece_id=XS7IKT3A6KJKML2W3EYWCLSSARZ6FFJ4KDWZLUCKTURFOL2ZOBSA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.459131301 (60h17m59.649429881s) [noise=true , address=45.91.116.9:4263 , piece_id=LAZ2KZJSCY6GARMUTRZ2RVICXHYCX2ZTRBPJWDEKCPN2HERSIW6A, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.462288615 (60h17m59.652587206s) [noise=true , address=104.36.82.13:4187 , piece_id=ODDCFSELQXRYXN2AZMB2KNXXUV63P2YD6RJ23MS4QNXB5IEO7DNQ, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.462765174 (60h17m59.65306376s) [noise=true , address=77.90.31.13:4627 , piece_id=R4MDCR3XY5CS5U2LOD6JDU3O54QPCY3GIYRNAT55ZFTXGCIQHBPQ, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.474108366 (60h17m59.664406952s) [noise=true , address=104.36.82.12:4149 , piece_id=KOVPCCPXD2J5GQQRUDA3ZDLN3YXBOE52FKHETBFJ6RBB3KLT4TQA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:28.474694567 (60h17m59.664993156s) [noise=true , address=45.91.116.4:4937 , piece_id=AHVM2B62U6HI4H4N4PSMT2SHDCTXDHOPHN3OEGF2A7GIYKGYEYMA, seg_pos={0 0}, upload=57807]: Uploading piece... Jul 26 23:53:29.067562207 (60h18m0.257860787s) [upload=57807]: Next returned nil segment Jul 26 23:53:29.067579570 (60h18m0.257878152s) [upload=57807]: Waiting for error group managing segments... Jul 26 23:53:29.201290526 (60h18m0.391589105s) [noise=true , address=45.91.116.8:4048 , piece_id=GJ5YGAXPFFB427XM6XRWVLR4JKVYD5OJ3EBNYBKBXQB5MMRRI2VA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.206214953 (60h18m0.396513522s) [noise=true , address=45.91.116.9:4261 , piece_id=LVPGW7GYIEPKASCMVDJ2R4PPKYEZEHIPS253HTHY76XXRJW3SY7Q, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.206901537 (60h18m0.397200102s) [noise=true , address=45.91.116.5:4298 , piece_id=Q4LX4S6G5JMMTIFO7IRF3RO6CFDGIM2GYXELYSEK327QH6XMI3FA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.208751292 (60h18m0.399049865s) [noise=true , address=45.91.116.12:4225 , piece_id=JA2M4UJRTF2HLVYK66NBLM6TN5LZO42OMOFJ4WIWELWG6A3LD6QA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.211295858 (60h18m0.401594438s) [noise=true , address=45.91.116.5:4297 , piece_id=HWEF7KAKGUO5NMJEFUQSZ7M7Y4JYMUVYQMSIFEPIBJT5M3QAEIDQ, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.223290823 (60h18m0.413589393s) [noise=true , address=45.91.116.10:4083 , piece_id=XS7IKT3A6KJKML2W3EYWCLSSARZ6FFJ4KDWZLUCKTURFOL2ZOBSA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.224169791 (60h18m0.414468363s) [noise=true , address=77.90.31.11:4555 , piece_id=KQ6B65YJMNGGH4BFEVXJL4O6ETZGUG37INGHXAYCEFW7XDK72FFQ, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.232858676 (60h18m0.423157249s) [noise=true , address=45.91.116.4:4937 , piece_id=AHVM2B62U6HI4H4N4PSMT2SHDCTXDHOPHN3OEGF2A7GIYKGYEYMA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.233891565 (60h18m0.424190138s) [noise=true , address=77.90.31.10:4520 , piece_id=XTGECWA4U3HUJZOZKS737KV64T3E6ZNFVQFC24IU7Q6TDLZXMR6Q, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:29.343280898 (60h18m0.533579472s) [noise=true , address=45.91.116.9:4263 , piece_id=LAZ2KZJSCY6GARMUTRZ2RVICXHYCX2ZTRBPJWDEKCPN2HERSIW6A, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:30.598304766 (60h18m1.788603339s) [noise=true , address=77.90.31.13:4627 , piece_id=R4MDCR3XY5CS5U2LOD6JDU3O54QPCY3GIYRNAT55ZFTXGCIQHBPQ, seg_pos={0 0}, upload=57807]: Done uploading piece. err: <nil> Jul 26 23:53:58.076525419 (60h18m29.266823985s) [noise=true , address=104.36.82.3:4765 , piece_id=NJPLIIP2OI6TAYPXNBL5ZPLMTNQTTPQHVOUVNRDA77JVNTDMFXRA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 139.59.144.22:50518->104.36.82.3:4765: use of closed network connection; write tcp 139.59.144.22:50518->104.36.82.3:4765: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50518->104.36.82.3:4765: use of closed network connection Jul 26 23:53:58.076592397 (60h18m29.266890966s) [noise=true , address=104.36.82.13:4187 , piece_id=ODDCFSELQXRYXN2AZMB2KNXXUV63P2YD6RJ23MS4QNXB5IEO7DNQ, seg_pos={0 0}, upload=57807]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:43544->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:43544->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43544->104.36.82.13:4187: use of closed network connection Jul 26 23:53:58.076624527 (60h18m29.266923106s) [noise=true , address=104.36.82.12:4149 , piece_id=KOVPCCPXD2J5GQQRUDA3ZDLN3YXBOE52FKHETBFJ6RBB3KLT4TQA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 139.59.144.22:46982->104.36.82.12:4149: use of closed network connection; write tcp 139.59.144.22:46982->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46982->104.36.82.12:4149: use of closed network connection Jul 26 23:53:58.076676231 (60h18m29.266974803s) [noise=true , address=104.36.82.3:4767 , piece_id=OCULMRJRZCGYVA3BOHJ2SE754TI3BD4IAPIRC3F5BOLQRITCSMBQ, seg_pos={0 0}, upload=57807]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:43756->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:43756->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43756->104.36.82.3:4767: use of closed network connection Jul 26 23:53:58.076687075 (60h18m29.266985645s) [noise=true , address=104.36.82.8:4971 , piece_id=3WEYFTBCZAOXS2EUJXGJCF2U6GJLNG6EHB77FNPTC5RTC2I3SFVA, seg_pos={0 0}, upload=57807]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:50138->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:50138->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50138->104.36.82.8:4971: use of closed network connection Jul 26 23:53:58.076723251 (60h18m29.267022097s) [seg_pos={0 0}, upload=57807]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:53:58.180562800 (60h18m29.370861367s) [upload=57807]: Done uploading segments... Jul 26 23:53:58.180590687 (60h18m29.370889252s) [upload=57807]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.105627341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX..] duration: 23.338487364s FAIL (2025-07-26 23:53:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 23.33871005s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [........XX....X....X.X..X........XX.X....X......X..X........] duration: 28.301929073s FAIL (2025-07-26 23:53:46 UTC) orbiter warning: context deadline exceeded Duration: 28.301956749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX..X..XXXXX....XX.XX..X.......X......X..XXX....XXX...X....] duration: 2.738846874s FAIL (2025-07-26 23:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.73887917s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [.X.XXXXXXXXX.XX..X.XXXXXXXX..X.....XX.X.X......X..X...X.X.X.] duration: 51.878490425s FAIL (2025-07-26 23:53:22 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=57760]: Upload finished. err: <nil> Jul 26 23:51:21.120844821 (60h15m52.311143392s) [upload=57777]: Starting upload Jul 26 23:51:21.120914819 (60h15m52.311213398s) [upload=57777]: Done starting upload Jul 26 23:51:21.121114266 (60h15m52.311412834s) [upload=57777]: Uploading segments... Jul 26 23:51:21.121315062 (60h15m52.311613645s) [upload=57777]: Got next segment. Inline: false Jul 26 23:51:21.121373173 (60h15m52.311671755s) [upload=57777]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:51:21.121393948 (60h15m52.31169253s) [upload=57777]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:51:21.318613089 (60h15m52.508911668s) [seg_pos={0 0}, upload=57777]: Begin upload segment... Jul 26 23:51:21.318678238 (60h15m52.508976817s) [seg_pos={0 0}, upload=57777]: Done begin upload segment. Jul 26 23:51:21.318709310 (60h15m52.509007893s) [noise=true , address=77.90.31.16:4737 , piece_id=74J35KLDFKHQEDFC5P5UWXF2CQNMGB5TETBQ2DY7OGEPERC5FJAQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.318772604 (60h15m52.509074513s) [noise=true , address=79.127.201.215:4357 , piece_id=H7WBF5RSI5VC6LHJ6KRPFEDP46V2WBY32QDIOHA6OW4SAEGYAR2Q, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.319158495 (60h15m52.509457077s) [noise=true , address=77.90.31.18:4823 , piece_id=IFSA66FBFC6X6GXBPEDSPFMCOQJR4X7CYAQXEDVDE5BYLTTEPK2A, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.319526702 (60h15m52.509825281s) [noise=true , address=104.36.82.14:4228 , piece_id=6F5FHJHJKWLDZZ7BJK5PTU4OAWLVVGKDYYSCRRA46P5CPD2P6VWQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.319890417 (60h15m52.510188998s) [noise=true , address=104.36.82.46:4516 , piece_id=AZIVYGF7HGC3U2DYRF7GIXYBAY3JW352FHL6IME6F6IYOA2QQTZA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.320323126 (60h15m52.510621718s) [noise=true , address=77.90.31.13:4648 , piece_id=U6TE7ZOCAW3CUVFRS3XW3FBLEC6DR6AY2SSFAEW6OS5CX3GAJV6Q, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.320812188 (60h15m52.511110776s) [noise=true , address=77.90.31.19:4867 , piece_id=FITJA6MPO6VWH2GJ47VAWJIJAZ3ZV6GK6WPLWOXE5M42GRMDWJUQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.321354390 (60h15m52.511652976s) [noise=true , address=104.36.82.3:4778 , piece_id=NLCDIH7LKKBHVB6PM2ZPGD7AQAGHQ5GF23D3CBMN3SWJDB3JMMPQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.321561681 (60h15m52.511860348s) [noise=true , address=104.36.82.54:4192 , piece_id=B7TUN6TFD6C54XAC6GXXLQJSBI3727647TS2K7ZGNW36U2J2HUHQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322119097 (60h15m52.512417683s) [noise=true , address=64.49.13.140:4508 , piece_id=35SDQJFL3BDYX7DTIF6YIP3JKUG6XCREWF5ETOZCPM3R25OM55UA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322487621 (60h15m52.512786208s) [noise=true , address=79.127.201.215:4353 , piece_id=XURH3ZG2VEEUOGYTV6IOLVNAD5G3SFFAYYJELKL6L4FLD4U6YWUA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322553663 (60h15m52.512852252s) [noise=true , address=79.127.219.38:4018 , piece_id=OMXH75H2YDMVFXSISNUI5EFQMWP6Z7IXV6E56BYIL5PPNCNWCXQQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.322855840 (60h15m52.513154423s) [noise=true , address=79.127.205.235:4685 , piece_id=O3LIB5E5AQ4GNEDUHLKPYWEXBCNKYS75EMT7PXCL26WZZSXTLXSA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.323201594 (60h15m52.513500174s) [noise=true , address=156.146.43.225:4434 , piece_id=XC66CSU3RHGMFIYYANZ2INZFOJ3VT5WO2KQWNIUKRCJPWENAKYAQ, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.323634016 (60h15m52.513932598s) [noise=true , address=104.36.82.53:4134 , piece_id=XRWBV5FUK5DZ24YLIZQWDTFWSC3XSM7YDTKYF3Y63N2DR6PYHEYA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.323866203 (60h15m52.514164791s) [noise=true , address=77.90.31.15:4714 , piece_id=RUZRS4S2WJENYK2NZWHRFXHFCRJCQ677IXXSQJUS4YO56R7GK2XA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.324366738 (60h15m52.514665321s) [noise=true , address=79.127.201.215:4360 , piece_id=XE4Q2S5TAXK7C5HYZMBZLJR66AEGKQSPYI3ZTDUEHTUOIX3GECAA, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.324716629 (60h15m52.515015218s) [noise=true , address=79.127.205.246:4284 , piece_id=QPRZOSHJFD272ITYE2KIIXNMHUCE6VCE6ARRTYVTZZVUU462MT7A, seg_pos={0 0}, upload=57777]: Uploading piece... Jul 26 23:51:21.880171655 (60h15m53.070470234s) [upload=57777]: Next returned nil segment Jul 26 23:51:21.880192137 (60h15m53.070490717s) [upload=57777]: Waiting for error group managing segments... Jul 26 23:51:22.070866647 (60h15m53.261165222s) [noise=true , address=77.90.31.16:4737 , piece_id=74J35KLDFKHQEDFC5P5UWXF2CQNMGB5TETBQ2DY7OGEPERC5FJAQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.131675699 (60h15m53.321974281s) [noise=true , address=79.127.219.38:4018 , piece_id=OMXH75H2YDMVFXSISNUI5EFQMWP6Z7IXV6E56BYIL5PPNCNWCXQQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.337500020 (60h15m53.527798604s) [noise=true , address=79.127.201.215:4357 , piece_id=H7WBF5RSI5VC6LHJ6KRPFEDP46V2WBY32QDIOHA6OW4SAEGYAR2Q, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.655708148 (60h15m53.846006715s) [noise=true , address=77.90.31.15:4714 , piece_id=RUZRS4S2WJENYK2NZWHRFXHFCRJCQ677IXXSQJUS4YO56R7GK2XA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: storage node overloaded, request limit: 2000 Jul 26 23:51:22.669727155 (60h15m53.860025739s) [noise=true , address=79.127.205.235:4685 , piece_id=O3LIB5E5AQ4GNEDUHLKPYWEXBCNKYS75EMT7PXCL26WZZSXTLXSA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.692515988 (60h15m53.882814634s) [noise=true , address=79.127.205.246:4284 , piece_id=QPRZOSHJFD272ITYE2KIIXNMHUCE6VCE6ARRTYVTZZVUU462MT7A, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:22.927479731 (60h15m54.117778296s) [noise=true , address=64.49.13.140:4508 , piece_id=35SDQJFL3BDYX7DTIF6YIP3JKUG6XCREWF5ETOZCPM3R25OM55UA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:23.572960259 (60h15m54.763258828s) [noise=true , address=156.146.43.225:4434 , piece_id=XC66CSU3RHGMFIYYANZ2INZFOJ3VT5WO2KQWNIUKRCJPWENAKYAQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:35.403823526 (60h16m6.594122095s) [noise=true , address=77.90.31.19:4867 , piece_id=FITJA6MPO6VWH2GJ47VAWJIJAZ3ZV6GK6WPLWOXE5M42GRMDWJUQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:37.584576677 (60h16m8.77487526s) [noise=true , address=79.127.201.215:4360 , piece_id=XE4Q2S5TAXK7C5HYZMBZLJR66AEGKQSPYI3ZTDUEHTUOIX3GECAA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:37.867812853 (60h16m9.058111422s) [noise=true , address=79.127.201.215:4353 , piece_id=XURH3ZG2VEEUOGYTV6IOLVNAD5G3SFFAYYJELKL6L4FLD4U6YWUA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:48.553560938 (60h16m19.743859517s) [noise=true , address=104.36.82.3:4778 , piece_id=NLCDIH7LKKBHVB6PM2ZPGD7AQAGHQ5GF23D3CBMN3SWJDB3JMMPQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:51:49.779710791 (60h16m20.970009371s) [noise=true , address=104.36.82.54:4192 , piece_id=B7TUN6TFD6C54XAC6GXXLQJSBI3727647TS2K7ZGNW36U2J2HUHQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:52:09.765619693 (60h16m40.955918273s) [noise=true , address=104.36.82.46:4516 , piece_id=AZIVYGF7HGC3U2DYRF7GIXYBAY3JW352FHL6IME6F6IYOA2QQTZA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:52:11.635776353 (60h16m42.826074922s) [noise=true , address=104.36.82.53:4134 , piece_id=XRWBV5FUK5DZ24YLIZQWDTFWSC3XSM7YDTKYF3Y63N2DR6PYHEYA, seg_pos={0 0}, upload=57777]: Done uploading piece. err: <nil> Jul 26 23:52:11.635805183 (60h16m42.826103762s) [seg_pos={0 0}, upload=57777]: Segment reached optimal threshold of 14 pieces. Jul 26 23:52:11.636019615 (60h16m42.826318195s) [noise=true , address=77.90.31.13:4648 , piece_id=U6TE7ZOCAW3CUVFRS3XW3FBLEC6DR6AY2SSFAEW6OS5CX3GAJV6Q, seg_pos={0 0}, upload=57777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh): protocol: expected piece hash; context canceled; EOF Jul 26 23:52:11.636039654 (60h16m42.826338233s) [noise=true , address=104.36.82.14:4228 , piece_id=6F5FHJHJKWLDZZ7BJK5PTU4OAWLVVGKDYYSCRRA46P5CPD2P6VWQ, seg_pos={0 0}, upload=57777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF): protocol: expected piece hash; context canceled; EOF Jul 26 23:52:11.636194965 (60h16m42.826493545s) [noise=true , address=77.90.31.18:4823 , piece_id=IFSA66FBFC6X6GXBPEDSPFMCOQJR4X7CYAQXEDVDE5BYLTTEPK2A, seg_pos={0 0}, upload=57777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: context canceled; context canceled Jul 26 23:52:11.636214607 (60h16m42.826513174s) [seg_pos={0 0}, upload=57777]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 23:52:11.636289661 (60h16m42.826588241s) [upload=57777]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:52:11.636294984 (60h16m42.826593563s) [upload=57777]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:52:11.791908693 (60h16m42.982207263s) [upload=57777]: Done uploading segments... Jul 26 23:52:11.791928887 (60h16m42.982227457s) [upload=57777]: Upload finished. err: <nil> Duration: 51.878528209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX..X..XXXXX....XX.XX..X.......X......X..XXX....XXX...X...] duration: 27.037453737s FAIL (2025-07-26 23:53:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.037491956s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X........XX....X....X.X..X........XX.X....X......X..X.......] duration: 30.157013995s FAIL (2025-07-26 23:52:52 UTC) orbiter warning: context deadline exceeded Duration: 30.157030645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX.XXXXX..XXXXX..XXXXX.] duration: 4.410915314s FAIL (2025-07-26 23:52:47 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 4.410959487s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXX..X.X.X.] duration: 26.73732772s FAIL (2025-07-26 23:52:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== JSXY6PSVGWNBCZSGL2BV3HKRRQ, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.034583128 (60h15m45.224881718s) [noise=true , address=45.91.116.5:4297 , piece_id=TGK4PRVEZNA244NF4ZTNV5C6HDOXC3CE56GVK2VEMCX3FH2FXKNQ, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.034912109 (60h15m45.225210699s) [noise=true , address=77.90.31.19:4842 , piece_id=WBEI3SPJ77HHAWOUFE5NMA6D4HSZCMACV33CJHC34FMJL7Q6MDAA, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.035263102 (60h15m45.225561693s) [noise=true , address=77.90.31.21:4877 , piece_id=75DD7UALF2CHB4GVZBU6SUCESY5YGNEYB3RO5VZBZZDYK7GEH5WQ, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.035620885 (60h15m45.225919473s) [noise=true , address=104.36.82.3:4767 , piece_id=UILT4IG6HP5TYYDGQENXPU5VYN3ENZESQYMNCORJYTF24DFO47QQ, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.035925995 (60h15m45.226224584s) [noise=true , address=77.90.31.14:4662 , piece_id=M5XY3UXFRKS7FJTGFID6STSSPAEQ24TMG7WC55MWS6PWHIY2TK7Q, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.036263008 (60h15m45.226561579s) [noise=true , address=104.36.82.11:4114 , piece_id=OAYGYKZRKS3XQLJ25MMLBUESZLK4YTP7TPB7YMBE4NGMC3THH3RQ, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.036606019 (60h15m45.226904607s) [noise=true , address=104.36.82.14:4223 , piece_id=RU223PR3PGQBR42CFFIQHTYQ6VWN36OD6QUNQSIAJVA6MGOG7JCA, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.036987456 (60h15m45.227286045s) [noise=true , address=104.36.82.10:4077 , piece_id=FLZ57A3GYXYXZLKPAD7Q5E7JV3KQTN4FKMI64LMSVNU6X6YKRATQ, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.037346834 (60h15m45.227645425s) [noise=true , address=104.36.82.13:4187 , piece_id=LJMHI4KCK7EUWWHNT2BGXMJOBF3DYXRQF6OR2MD6BR4VD52FNBMA, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.037707994 (60h15m45.228006583s) [noise=true , address=45.91.116.8:4048 , piece_id=WHX5YC255U2LRRRQ3H3JSJK7ZX77RYRBRAC44OQNIAP4ELIWSI4Q, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.038113874 (60h15m45.228412465s) [noise=true , address=45.91.116.4:4938 , piece_id=NARGB2MXS3QHLIYFUAPIFG6ZY2574WLJ2SYUIFJNETSCDVK5JXUA, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.038241809 (60h15m45.228540399s) [noise=true , address=104.36.82.13:4185 , piece_id=4PFQOPFPQ5SIZWW5C5JKON4SA7SZPEQYYIUPN4YUIHSPPFPQ6RUQ, seg_pos={0 0}, upload=57774]: Uploading piece... Jul 26 23:51:14.644369886 (60h15m45.834668469s) [upload=57774]: Next returned nil segment Jul 26 23:51:14.644389486 (60h15m45.83468807s) [upload=57774]: Waiting for error group managing segments... Jul 26 23:51:14.780869163 (60h15m45.971167747s) [noise=true , address=45.91.116.8:4048 , piece_id=WHX5YC255U2LRRRQ3H3JSJK7ZX77RYRBRAC44OQNIAP4ELIWSI4Q, seg_pos={0 0}, upload=57774]: Done uploading piece. err: <nil> Jul 26 23:51:14.784200911 (60h15m45.974499484s) [noise=true , address=77.90.31.14:4662 , piece_id=M5XY3UXFRKS7FJTGFID6STSSPAEQ24TMG7WC55MWS6PWHIY2TK7Q, seg_pos={0 0}, upload=57774]: Done uploading piece. err: <nil> Jul 26 23:51:14.791901910 (60h15m45.982200502s) [noise=true , address=77.90.31.21:4877 , piece_id=75DD7UALF2CHB4GVZBU6SUCESY5YGNEYB3RO5VZBZZDYK7GEH5WQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: <nil> Jul 26 23:51:14.796679598 (60h15m45.986978168s) [noise=true , address=45.91.116.6:4975 , piece_id=FF3KJD6YQIOJ2NXYGB4EZ6IJ3QTGSCDAPDGBPZZQUYY6OECT3Z7A, seg_pos={0 0}, upload=57774]: Done uploading piece. err: <nil> Jul 26 23:51:14.813805688 (60h15m46.004104275s) [noise=true , address=45.91.116.4:4938 , piece_id=NARGB2MXS3QHLIYFUAPIFG6ZY2574WLJ2SYUIFJNETSCDVK5JXUA, seg_pos={0 0}, upload=57774]: Done uploading piece. err: <nil> Jul 26 23:51:14.828142614 (60h15m46.018441179s) [noise=true , address=45.91.116.5:4297 , piece_id=TGK4PRVEZNA244NF4ZTNV5C6HDOXC3CE56GVK2VEMCX3FH2FXKNQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: <nil> Jul 26 23:51:16.235787954 (60h15m47.426086526s) [noise=true , address=77.90.31.19:4842 , piece_id=WBEI3SPJ77HHAWOUFE5NMA6D4HSZCMACV33CJHC34FMJL7Q6MDAA, seg_pos={0 0}, upload=57774]: Done uploading piece. err: <nil> Jul 26 23:51:43.656063073 (60h16m14.846361652s) [noise=true , address=104.36.82.6:4875 , piece_id=QNTKMERNWCARCSAHNGMPXXYD2WJSXY6PSVGWNBCZSGL2BV3HKRRQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 139.59.144.22:42440->104.36.82.6:4875: use of closed network connection; write tcp 139.59.144.22:42440->104.36.82.6:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42440->104.36.82.6:4875: use of closed network connection Jul 26 23:51:43.656113714 (60h16m14.846412294s) [noise=true , address=104.36.82.3:4767 , piece_id=UILT4IG6HP5TYYDGQENXPU5VYN3ENZESQYMNCORJYTF24DFO47QQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:57134->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:57134->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57134->104.36.82.3:4767: use of closed network connection Jul 26 23:51:43.656151773 (60h16m14.846450342s) [noise=true , address=104.36.82.16:4294 , piece_id=ET2DST4DXQVCN5L3STOL3WL2TJWJ2VN2MKF77ZXEK6VMNV7F5DAQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:52032->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:52032->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52032->104.36.82.16:4294: use of closed network connection Jul 26 23:51:43.656219541 (60h16m14.846518111s) [noise=true , address=104.36.82.14:4223 , piece_id=RU223PR3PGQBR42CFFIQHTYQ6VWN36OD6QUNQSIAJVA6MGOG7JCA, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 139.59.144.22:46166->104.36.82.14:4223: use of closed network connection; write tcp 139.59.144.22:46166->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46166->104.36.82.14:4223: use of closed network connection Jul 26 23:51:43.656290042 (60h16m14.84658861s) [noise=true , address=104.36.82.10:4077 , piece_id=FLZ57A3GYXYXZLKPAD7Q5E7JV3KQTN4FKMI64LMSVNU6X6YKRATQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 139.59.144.22:60134->104.36.82.10:4077: use of closed network connection; write tcp 139.59.144.22:60134->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60134->104.36.82.10:4077: use of closed network connection Jul 26 23:51:43.656348027 (60h16m14.846646607s) [noise=true , address=104.36.82.12:4150 , piece_id=W524RYZWCH6IJ5HEX42BVXJYW3TY3TUJ3HQ4F2ENPITTJZGQQLOA, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:57012->104.36.82.12:4150: use of closed network connection; context deadline exceeded Jul 26 23:51:43.656358220 (60h16m14.84665679s) [noise=true , address=104.36.82.11:4114 , piece_id=OAYGYKZRKS3XQLJ25MMLBUESZLK4YTP7TPB7YMBE4NGMC3THH3RQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:38654->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:38654->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38654->104.36.82.11:4114: use of closed network connection Jul 26 23:51:43.656369751 (60h16m14.846668324s) [noise=true , address=104.36.82.13:4187 , piece_id=LJMHI4KCK7EUWWHNT2BGXMJOBF3DYXRQF6OR2MD6BR4VD52FNBMA, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:56232->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:56232->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56232->104.36.82.13:4187: use of closed network connection Jul 26 23:51:43.656377687 (60h16m14.846676261s) [noise=true , address=104.36.82.13:4185 , piece_id=4PFQOPFPQ5SIZWW5C5JKON4SA7SZPEQYYIUPN4YUIHSPPFPQ6RUQ, seg_pos={0 0}, upload=57774]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:45906->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:45906->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45906->104.36.82.13:4185: use of closed network connection Jul 26 23:51:43.656409212 (60h16m14.846707803s) [seg_pos={0 0}, upload=57774]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:51:43.760583060 (60h16m14.950881634s) [upload=57774]: Done uploading segments... Jul 26 23:51:43.760611169 (60h16m14.950909749s) [upload=57774]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 26.73736444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X........XX....X....X.X..X........XX.X....X......X..X......] duration: 56.851583344s FAIL (2025-07-26 23:52:22 UTC) orbiter warning: context deadline exceeded Duration: 56.851662659s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)