Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X...X..............X...........................XXXXX....] duration: 12.929553883s FAIL (2025-08-01 22:00:29 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 12.929578186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 2.668113091s FAIL (2025-08-01 22:00:17 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.668160875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration: 3.921924561s FAIL (2025-08-01 22:00:17 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.921964639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 7.232847594s FAIL (2025-08-01 22:00:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.232869547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.230747322s FAIL (2025-08-01 22:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.230773075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X...X..............X...........................XXXXX...] duration: 23.93671876s FAIL (2025-08-01 21:59:55 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 23.936744334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X...........................................X............] duration: 12.165814895s FAIL (2025-08-01 21:59:44 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 12.16584025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 3.662953233s FAIL (2025-08-01 21:59:35 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.662978225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 3.009673117s FAIL (2025-08-01 21:59:24 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.009693054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 4.931627171s FAIL (2025-08-01 21:59:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.931661323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.78795829s FAIL (2025-08-01 21:59:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.788575176s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 2.713854187s FAIL (2025-08-01 21:58:59 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.713883007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X...........................................X...........] duration: 3.589618519s FAIL (2025-08-01 21:58:54 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 3.589646773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X...X..............X...........................XXXXX..] duration: 8.516770709s FAIL (2025-08-01 21:58:39 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 8.51680021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 2.602711118s FAIL (2025-08-01 21:58:29 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.602742363s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.765860421s FAIL (2025-08-01 21:58:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.765882826s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 4.548453712s FAIL (2025-08-01 21:58:22 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 4.548478104s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 4.030366915s FAIL (2025-08-01 21:58:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.030381609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.000646678s FAIL (2025-08-01 21:58:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000659749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 2.768047303s FAIL (2025-08-01 21:58:09 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.76818142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X...X..............X...........................XXXXX.] duration: 8.011326117s FAIL (2025-08-01 21:57:59 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 8.011360914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........................................X..........] duration: 3.561343115s FAIL (2025-08-01 21:57:59 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 3.561375946s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.677320286s FAIL (2025-08-01 21:57:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.677343495s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 2.38329285s FAIL (2025-08-01 21:57:48 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.383331475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration: 12.774844364s FAIL (2025-08-01 21:57:31 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 12.774885184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.109123546s FAIL (2025-08-01 21:57:18 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.109135155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X...X..............X...........................XXXXX] duration: 30.114936401s FAIL (2025-08-01 21:57:14 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 30.114958516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.020517202s FAIL (2025-08-01 21:57:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.020535515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X...........................................X.........] duration: 815.487412ms FAIL (2025-08-01 21:57:04 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 815.514656ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 2.29091444s FAIL (2025-08-01 21:56:59 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.290945773s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.993019017s FAIL (2025-08-01 21:56:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.99304421s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 2.353983991s FAIL (2025-08-01 21:56:37 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.35400864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........................................X........] duration: 831.241572ms FAIL (2025-08-01 21:56:27 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 831.2605ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X..............X...........................XXXX] duration: 30.10748661s FAIL (2025-08-01 21:56:25 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 6) Duration: 30.107512014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.115123425s FAIL (2025-08-01 21:56:18 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== iece... Aug 1 21:55:18.797182881 (51h58m37.84620174s) [noise=true , address=45.91.116.11:4130 , piece_id=P5FNLXIZ7HYKI6F4FW7E7NDF4AYWUJ7LIAYJPPY4O2EINEPKH3GA, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.797523834 (51h58m37.846542697s) [noise=true , address=104.36.82.12:4171 , piece_id=GD3CRG2U3E5UGGJZR7XZ5MYBBSXIJZFZGKWQG3ZSWTLCNDDN32BQ, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.797757715 (51h58m37.846776577s) [noise=true , address=64.49.13.137:4499 , piece_id=SE3PJCWKLDUZATVSLBAXBNZLIBVGHVBSX2VEXV2AB6IOFU27VIXA, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.798068893 (51h58m37.847087752s) [noise=true , address=104.36.82.53:4147 , piece_id=7WDJQLTXQRC3444XNVFGT7RQIAER2KK3FH2NK46IWJWIHPDDSLYA, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.798334434 (51h58m37.847353293s) [noise=true , address=104.36.82.62:4423 , piece_id=BR5XDKLWZIJR6CIIZOEKSROO7BIYSNOMDSJBQZQQVWOIY3N6RXJQ, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.798555994 (51h58m37.847574853s) [noise=true , address=104.36.82.43:4854 , piece_id=J44OQAD4VBP75LKM6DMKHWTDQRIKXLAAVHAOHRL4KNEFWCGMIAOQ, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.798778383 (51h58m37.847797243s) [noise=true , address=104.36.82.13:4216 , piece_id=KGEEC7OU2GOHIE6BG3B3NIDNPI3IUE3BIZNVPTAZVYYK6NEVPYMQ, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.799046701 (51h58m37.84806556s) [noise=true , address=104.36.82.42:4799 , piece_id=PIP3N73GJC3SXIHKW4A53JFQ3DUM56DWLHISMKJLOXDB4VHX6D3A, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.799314618 (51h58m37.848333482s) [noise=true , address=77.90.31.16:4768 , piece_id=I7IGBVHP4WPL5Q4T5ZKPVFWF4XAWIWSSMU53JDPKAFXWGZX5I26Q, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:18.799634075 (51h58m37.848652934s) [noise=true , address=104.36.82.5:4872 , piece_id=Y33EOEAFRHWGX6FAFHCU7UTHMVGINWUQSLQFMMDNEUUS7GM5NGOA, seg_pos={0 0}, upload=49956]: Uploading piece... Aug 1 21:55:19.397109310 (51h58m38.446128186s) [upload=49956]: Next returned nil segment Aug 1 21:55:19.397132393 (51h58m38.44615126s) [upload=49956]: Waiting for error group managing segments... Aug 1 21:55:19.570541077 (51h58m38.619559927s) [noise=true , address=45.91.116.11:4130 , piece_id=P5FNLXIZ7HYKI6F4FW7E7NDF4AYWUJ7LIAYJPPY4O2EINEPKH3GA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: <nil> Aug 1 21:55:19.572639225 (51h58m38.621658071s) [noise=true , address=77.90.31.16:4768 , piece_id=I7IGBVHP4WPL5Q4T5ZKPVFWF4XAWIWSSMU53JDPKAFXWGZX5I26Q, seg_pos={0 0}, upload=49956]: Done uploading piece. err: <nil> Aug 1 21:55:20.075876393 (51h58m39.12489524s) [noise=true , address=64.49.13.137:4499 , piece_id=SE3PJCWKLDUZATVSLBAXBNZLIBVGHVBSX2VEXV2AB6IOFU27VIXA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: <nil> Aug 1 21:56:18.275873176 (51h59m37.324892024s) [noise=true , address=104.36.82.52:4085 , piece_id=6E5AI22ULCMDBFF7S3KYY5QT3T4MGUENQGDKUKDK5BG34YFDMA6A, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: rpc: dial tcp 104.36.82.52:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.275950141 (51h59m37.324968997s) [noise=true , address=104.36.82.53:4147 , piece_id=7WDJQLTXQRC3444XNVFGT7RQIAER2KK3FH2NK46IWJWIHPDDSLYA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276001489 (51h59m37.325020345s) [noise=true , address=104.36.82.13:4216 , piece_id=KGEEC7OU2GOHIE6BG3B3NIDNPI3IUE3BIZNVPTAZVYYK6NEVPYMQ, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:12AgazvjebAo7K4NdQ7zsKkETiNQNfiykHcBDcmXej6eipwWpFu, address:104.36.82.13:4216): protocol: rpc: dial tcp 104.36.82.13:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276064701 (51h59m37.325083557s) [noise=true , address=104.36.82.42:4799 , piece_id=PIP3N73GJC3SXIHKW4A53JFQ3DUM56DWLHISMKJLOXDB4VHX6D3A, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:104.36.82.42:4799): protocol: rpc: dial tcp 104.36.82.42:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276115049 (51h59m37.325133906s) [noise=true , address=104.36.82.61:4396 , piece_id=2YPQKY6SG42ZEWQ3AM5SO653TORETFTAVNPNVA3G5PRI5NWYXBWA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:12XDCfMD1mxh7t5V5SHkwfFk4KYgPTj4qL2sgygCyVhVUF7FBXY, address:104.36.82.61:4396): protocol: rpc: dial tcp 104.36.82.61:4396: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276168088 (51h59m37.325186944s) [noise=true , address=104.36.82.10:4080 , piece_id=EK5H4QYC2PSXGNFOMFRDY22XFMIEQ7X6IL7RRMJOHKMZA4GXSHCA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3, address:104.36.82.10:4080): protocol: rpc: dial tcp 104.36.82.10:4080: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276214935 (51h59m37.325233792s) [noise=true , address=104.36.82.5:4872 , piece_id=Y33EOEAFRHWGX6FAFHCU7UTHMVGINWUQSLQFMMDNEUUS7GM5NGOA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: rpc: dial tcp 104.36.82.5:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276265503 (51h59m37.325284358s) [noise=true , address=104.36.82.43:4854 , piece_id=J44OQAD4VBP75LKM6DMKHWTDQRIKXLAAVHAOHRL4KNEFWCGMIAOQ, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:12eitcwvcNX2GyMiv4GdfDD8wmgb6a1d3ggaGVxbpyKAjPjshmV, address:104.36.82.43:4854): protocol: rpc: dial tcp 104.36.82.43:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276310233 (51h59m37.325329089s) [noise=true , address=104.36.82.12:4171 , piece_id=GD3CRG2U3E5UGGJZR7XZ5MYBBSXIJZFZGKWQG3ZSWTLCNDDN32BQ, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:1z2Fh5fPFTb9VZX6to5hdUYUGst63Ti1BVtdVtYk7vkrUqXMKM, address:104.36.82.12:4171): protocol: rpc: dial tcp 104.36.82.12:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276359799 (51h59m37.325378655s) [noise=true , address=104.36.82.47:4529 , piece_id=ZZMIPHCAVN25M326DDIEB2X5D7Z7TIVKGZBRT2OP2D7BQMHTZVYQ, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: rpc: dial tcp 104.36.82.47:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276390265 (51h59m37.32540912s) [noise=true , address=104.36.82.62:4423 , piece_id=BR5XDKLWZIJR6CIIZOEKSROO7BIYSNOMDSJBQZQQVWOIY3N6RXJQ, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:12fUb7FPBapbEwdkCE7vsgwcbF5B8KGcDVZAtihDB9SDuvUpzA5, address:104.36.82.62:4423): protocol: rpc: dial tcp 104.36.82.62:4423: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276415399 (51h59m37.325434255s) [noise=true , address=104.36.82.50:4044 , piece_id=6K4XBPCMAT5PPHSZK6JR66EFAUOJLVQEYMMX6SBIE3W2WXJHMGKA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:1oEaCgnsStJePrKHSXLLmwsAuYPEmgFbejFAMmNMqLp8m8vxiC, address:104.36.82.50:4044): protocol: write tcp 139.59.144.22:41116->104.36.82.50:4044: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276448816 (51h59m37.325467671s) [noise=true , address=104.36.82.4:4811 , piece_id=6CV76ZDH7Z57VNUZCOIHP6E2C45HRYK5SGQKDTQQLJYVJEDU6HGA, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276461074 (51h59m37.325479922s) [noise=true , address=104.36.82.16:4308 , piece_id=UKWMOVFTO2LWLRRYZBT2XB5HI2EL7KVCAZ2B7CO2VBG53XLRKSHQ, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:104.36.82.16:4308): protocol: rpc: dial tcp 104.36.82.16:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276467271 (51h59m37.325486127s) [noise=true , address=104.36.82.15:4289 , piece_id=A6QIWFOOHUJ4RUSXTSHHFLTKCE4MWM2CPEIE4OWPWX4BOE7GIXNQ, seg_pos={0 0}, upload=49956]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: write tcp 139.59.144.22:38484->104.36.82.15:4289: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:18.276508416 (51h59m37.325527278s) [seg_pos={0 0}, upload=49956]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Aug 1 21:56:18.388511539 (51h59m37.437530399s) [upload=49956]: Done uploading segments... Aug 1 21:56:18.388557135 (51h59m37.437575995s) [upload=49956]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.115139225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.118546307s FAIL (2025-08-01 21:56:16 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.118598081s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001169417s FAIL (2025-08-01 21:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001181164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X...........................................X.......] duration: 1.496648223s FAIL (2025-08-01 21:56:10 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.496673828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001462467s FAIL (2025-08-01 21:56:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001480487s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.135080222s FAIL (2025-08-01 21:55:18 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 9940]: Uploading piece... Aug 1 21:54:18.451664752 (51h57m37.500683611s) [noise=true , address=104.36.82.10:4082 , piece_id=DKCJ2ZCPVTESQZ6RBQWXDK6EIYRKDSP5EIFIGKMSZBXJGDRY3JZA, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:18.451889081 (51h57m37.500907938s) [noise=true , address=104.36.82.23:4469 , piece_id=ZOSRD3BZ53QK46VRTIKGGUM5WOAQUIGQOONYMB657SSCKVUCD2LQ, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:18.451981405 (51h57m37.501000268s) [noise=true , address=104.36.82.6:4894 , piece_id=KP3HYHY4QLEQBIP65USZMZJJASNGQK6IF3QOC2ECNKL34IJXYVYA, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:18.452119341 (51h57m37.501138197s) [noise=true , address=104.36.82.44:4903 , piece_id=GHP6UU5IVRCJL5CJPREVIHXBCGN2C7F6LJECRQ4ZFMCLR5LWOQMQ, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:18.452472727 (51h57m37.501491598s) [noise=true , address=104.36.82.41:4763 , piece_id=V7CW7F5QUIYN56BQYB5C6OISAXEGK6IBQZTMH5H27JCCEV36FM5Q, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:18.454250604 (51h57m37.50326947s) [noise=true , address=104.36.82.58:4290 , piece_id=ATOGDCCYM2F4RB7SKIHBK4XVOHO7OPEWTKNI6THJADWI23QUA3KQ, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:18.454890136 (51h57m37.503909003s) [noise=true , address=104.36.82.47:4541 , piece_id=UP7QEEZEQST3M3OJHKOXEJYBWBC4JLYYPKGIDSPKCD2ETNZJZ24Q, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:18.456472873 (51h57m37.505491739s) [noise=true , address=79.127.220.98:4399 , piece_id=HIIQGWLJD73SD47H6TISHNWRUKIPNUB5S437EX67KUG7HG4Q7EEA, seg_pos={0 0}, upload=49940]: Uploading piece... Aug 1 21:54:19.180734088 (51h57m38.229752935s) [upload=49940]: Next returned nil segment Aug 1 21:54:19.180750136 (51h57m38.229768994s) [upload=49940]: Waiting for error group managing segments... Aug 1 21:54:19.383525027 (51h57m38.432543884s) [noise=true , address=79.127.220.98:4399 , piece_id=HIIQGWLJD73SD47H6TISHNWRUKIPNUB5S437EX67KUG7HG4Q7EEA, seg_pos={0 0}, upload=49940]: Done uploading piece. err: <nil> Aug 1 21:55:17.985257697 (51h58m37.034276551s) [noise=true , address=104.36.82.14:4245 , piece_id=TC2VROSOB6SOMACTSXFNTR6SEZANT6RKCGQPAU34XNG2ZAIB22UQ, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: rpc: dial tcp 104.36.82.14:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985308509 (51h58m37.034327366s) [noise=true , address=104.36.82.58:4290 , piece_id=ATOGDCCYM2F4RB7SKIHBK4XVOHO7OPEWTKNI6THJADWI23QUA3KQ, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1YieSs1jRT5hKVKc7DuME4rFeX2BmeNngjBaGoB2hRrNwiLakv, address:104.36.82.58:4290): protocol: rpc: dial tcp 104.36.82.58:4290: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985352784 (51h58m37.034371645s) [noise=true , address=104.36.82.11:4127 , piece_id=MRPKVVU3V2UMQ2SNVT3G34VUNKIFQGJYHLIF6C5NTYRFWOKGJNMA, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: rpc: dial tcp 104.36.82.11:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985368676 (51h58m37.034387532s) [noise=true , address=104.36.82.55:4198 , piece_id=TNPPG4VWBVW4MZYC3NP5TFILOHGAD6IEYEUUYL5XFBIPFK43ZB4Q, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: rpc: dial tcp 104.36.82.55:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985425381 (51h58m37.034444244s) [noise=true , address=104.36.82.61:4392 , piece_id=4K4NIHUJ6ZOBRXB2GOOMYAHNYTDUXHNUW45ETK4ZBYGB7F3PUZHQ, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk, address:104.36.82.61:4392): protocol: rpc: dial tcp 104.36.82.61:4392: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985454428 (51h58m37.034473285s) [noise=true , address=104.36.82.15:4289 , piece_id=SM32KE7VQ7X2B7ZICTTAA6RGKK72AIRUIOGZ6WV3EBAHOG3I2XKA, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985502994 (51h58m37.034521852s) [noise=true , address=104.36.82.6:4894 , piece_id=KP3HYHY4QLEQBIP65USZMZJJASNGQK6IF3QOC2ECNKL34IJXYVYA, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:12fZsj77XZ8CnYD6iwzEVKPn6yLyxWMtibzsRXcuGKwNxEXB6Hq, address:104.36.82.6:4894): protocol: rpc: dial tcp 104.36.82.6:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985534776 (51h58m37.034553632s) [noise=true , address=104.36.82.47:4541 , piece_id=UP7QEEZEQST3M3OJHKOXEJYBWBC4JLYYPKGIDSPKCD2ETNZJZ24Q, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985577443 (51h58m37.034596302s) [noise=true , address=104.36.82.41:4763 , piece_id=V7CW7F5QUIYN56BQYB5C6OISAXEGK6IBQZTMH5H27JCCEV36FM5Q, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:104.36.82.41:4763): protocol: rpc: dial tcp 104.36.82.41:4763: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985596661 (51h58m37.034615517s) [noise=true , address=104.36.82.3:4773 , piece_id=2Z53MSOVQCM4TJ2CG74TDVZDUZ6DVJZITSHLHRKMVGEPXK7Y373A, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:104.36.82.3:4773): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:55:17.985652365 (51h58m37.034671223s) [noise=true , address=104.36.82.37:4672 , piece_id=IQMNDFVN5U4KREW2Y655IBWDEKSLLPGDQ2Y4P4XNBSSEZSXXCTPQ, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1G9dUkPkLRrEkMGsTiaN9iB8YnZ9GF7moCUPhGiu1XDDi8XnSu, address:104.36.82.37:4672): protocol: write tcp 139.59.144.22:39088->104.36.82.37:4672: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985692071 (51h58m37.03471093s) [noise=true , address=104.36.82.4:4817 , piece_id=G4VK3ODL5Q55MLTDV4DZ5QWYXH54PFFQLF6AF5NKT5K5AN5GWOJQ, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz, address:104.36.82.4:4817): protocol: rpc: dial tcp 104.36.82.4:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985715656 (51h58m37.03473452s) [noise=true , address=104.36.82.23:4469 , piece_id=ZOSRD3BZ53QK46VRTIKGGUM5WOAQUIGQOONYMB657SSCKVUCD2LQ, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:57766->104.36.82.23:4469: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985723436 (51h58m37.034742279s) [noise=true , address=104.36.82.49:4000 , piece_id=LB2L7ZN43VYMPN62S2C7V4ZSWLM3DHMMAAYF5WKXVNTVRPTACP7Q, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:12chhfe1weiPC1jjv8vbScP7uFVNXFr6s3UcNRkH8gsSAtGD16D, address:104.36.82.49:4000): protocol: rpc: dial tcp 104.36.82.49:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985728216 (51h58m37.034747069s) [noise=true , address=104.36.82.51:4078 , piece_id=BVURAKHY33WFQT4T5TTU6JPZEHT3ALZNIGV5RGQ25ZDRGWGIRE5Q, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd, address:104.36.82.51:4078): protocol: rpc: dial tcp 104.36.82.51:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985755687 (51h58m37.034774548s) [noise=true , address=104.36.82.10:4082 , piece_id=DKCJ2ZCPVTESQZ6RBQWXDK6EIYRKDSP5EIFIGKMSZBXJGDRY3JZA, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: rpc: dial tcp 104.36.82.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985770629 (51h58m37.034789471s) [noise=true , address=104.36.82.44:4903 , piece_id=GHP6UU5IVRCJL5CJPREVIHXBCGN2C7F6LJECRQ4ZFMCLR5LWOQMQ, seg_pos={0 0}, upload=49940]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: rpc: dial tcp 104.36.82.44:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:17.985804940 (51h58m37.034823802s) [seg_pos={0 0}, upload=49940]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Aug 1 21:55:18.118139865 (51h58m37.167158738s) [upload=49940]: Done uploading segments... Aug 1 21:55:18.118171359 (51h58m37.167190229s) [upload=49940]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.135122352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X...X..............X...........................XXX] duration: 30.16198942s FAIL (2025-08-01 21:55:18 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 4) Duration: 30.162019172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.000616421s FAIL (2025-08-01 21:55:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000629859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........................................X......] duration: 844.709867ms FAIL (2025-08-01 21:55:01 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 844.803411ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.295900351s FAIL (2025-08-01 21:54:34 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 21:52:17.921392832 (51h55m36.970411696s) [upload=49907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:52:18.055386824 (51h55m37.104405689s) [upload=49907]: Done uploading segments... Aug 1 21:52:18.055422517 (51h55m37.104441366s) [upload=49907]: Upload finished. err: <nil> Aug 1 21:53:29.365104050 (51h56m48.414122906s) [upload=49927]: Starting upload Aug 1 21:53:29.365175879 (51h56m48.414194743s) [upload=49927]: Done starting upload Aug 1 21:53:29.365455201 (51h56m48.414474043s) [upload=49927]: Uploading segments... Aug 1 21:53:29.365486823 (51h56m48.414505686s) [upload=49927]: Got next segment. Inline: false Aug 1 21:53:29.365542702 (51h56m48.414561545s) [upload=49927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:53:29.365546365 (51h56m48.414565221s) [upload=49927]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:53:29.472662631 (51h56m48.52168149s) [seg_pos={0 0}, upload=49927]: Begin upload segment... Aug 1 21:53:29.472724832 (51h56m48.521743682s) [seg_pos={0 0}, upload=49927]: Done begin upload segment. Aug 1 21:53:29.472774947 (51h56m48.521793806s) [noise=true , address=45.91.116.8:4047 , piece_id=N3R4B6GHWNAX45LK6WJN5PKY3ISWXG3SEBM4Z2TOKISG3X4RWRUQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.472782672 (51h56m48.521801528s) [noise=true , address=104.36.82.5:4838 , piece_id=ACWYTC5NJWJTXLKVS4XF6HHAZBQ5FEIP6COOTEEZPCLJDA6CP7XQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473218746 (51h56m48.522237602s) [noise=true , address=45.91.116.10:4081 , piece_id=H25Q5MVZIXIILL7X6HXNFDKOSZVLXC4RML7YZJ4JTNCCKZUILM4A, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473262022 (51h56m48.52228088s) [noise=true , address=77.90.31.18:4805 , piece_id=4T72ITVUYQHKAF6F25SOF6UV7JO4Z5W6VLPBTE3U3BH5XNQMEQTA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473648227 (51h56m48.522667086s) [noise=true , address=45.91.116.6:4975 , piece_id=S72OHKFBDJWBNQ5W5HFPZ2NVGBMVERKOQAE3FNEAYL3CNDENPD3A, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473800893 (51h56m48.522819766s) [noise=true , address=104.36.82.5:4840 , piece_id=SIYOT6HC7KBXBR52G7CEB75QDYQVJMHUQZRQE4E7XX4XFKH3ETBA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474079503 (51h56m48.523098359s) [noise=true , address=77.90.31.21:4877 , piece_id=PPRGMYO2DNSDIH3CCE4DKJEKVKZQG6S4AG4QN3B5SF6J67KAGMIA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474256677 (51h56m48.523275534s) [noise=true , address=77.90.31.16:4733 , piece_id=JXEWLMPA3TTXHJROG2QAH7VFYLDO5WBAYJSHTG4ANSS3SL66BZ6A, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474428248 (51h56m48.523447107s) [noise=true , address=45.91.116.7:4010 , piece_id=QNAUYQ4R7KGAOPR63GSBWLU47ORBKUFQOL4D4WWS32KS3VSD4YAQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474718468 (51h56m48.523737324s) [noise=true , address=45.91.116.10:4082 , piece_id=DM2AG74TCYCXLPZDK2TDZ2QIBWMCWJGVSDEMS2ZV47SO4ZWSFZKQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.475166790 (51h56m48.524185646s) [noise=true , address=45.91.116.13:4155 , piece_id=ITXRUUXIZYQE3EIHOZ2YP4FAGUD2IYTSJLMWDL7XYK2J6JPHMH6Q, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.478494979 (51h56m48.527513844s) [noise=true , address=104.36.82.2:4729 , piece_id=AYCGSBJNIVV6ZHUYTIXGHMFF5S47JSA54K26UR25JTHCLTFJCEWA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.479015210 (51h56m48.528034066s) [noise=true , address=104.36.82.11:4114 , piece_id=LHYBE5QL6DJLSZNA47YGOF2ZQ6CSC4HRGHEG2GLQGLLMWEIHF3CQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.479563863 (51h56m48.528582735s) [noise=true , address=104.36.82.3:4765 , piece_id=X5IVSDQBVCYXODA3SNVMW27WY6MLUZPOA2RQEIHRFLOPZO5DVANQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.480035123 (51h56m48.529053986s) [noise=true , address=104.36.82.8:4970 , piece_id=RZLKNKN2PLZP44SK4XQ66IXYFK54UX4U3FQZG7BFXGDTUREKZYOA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.480380992 (51h56m48.529399835s) [noise=true , address=77.90.31.14:4662 , piece_id=7I44ITGD3VFOQYFY23XUUYJ3OA3DX3W77A3X5VUKF2HTJR7U7KQQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:30.090118150 (51h56m49.139137016s) [upload=49927]: Next returned nil segment Aug 1 21:53:30.090139229 (51h56m49.139158095s) [upload=49927]: Waiting for error group managing segments... Aug 1 21:53:30.231512679 (51h56m49.280531522s) [noise=true , address=77.90.31.16:4733 , piece_id=JXEWLMPA3TTXHJROG2QAH7VFYLDO5WBAYJSHTG4ANSS3SL66BZ6A, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.233188516 (51h56m49.282207364s) [noise=true , address=45.91.116.10:4082 , piece_id=DM2AG74TCYCXLPZDK2TDZ2QIBWMCWJGVSDEMS2ZV47SO4ZWSFZKQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.234293524 (51h56m49.283312395s) [noise=true , address=77.90.31.14:4662 , piece_id=7I44ITGD3VFOQYFY23XUUYJ3OA3DX3W77A3X5VUKF2HTJR7U7KQQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.236894742 (51h56m49.285913585s) [noise=true , address=45.91.116.7:4010 , piece_id=QNAUYQ4R7KGAOPR63GSBWLU47ORBKUFQOL4D4WWS32KS3VSD4YAQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.239401166 (51h56m49.28842002s) [noise=true , address=45.91.116.8:4047 , piece_id=N3R4B6GHWNAX45LK6WJN5PKY3ISWXG3SEBM4Z2TOKISG3X4RWRUQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.239827716 (51h56m49.288846564s) [noise=true , address=45.91.116.10:4081 , piece_id=H25Q5MVZIXIILL7X6HXNFDKOSZVLXC4RML7YZJ4JTNCCKZUILM4A, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.243422005 (51h56m49.292440857s) [noise=true , address=45.91.116.6:4975 , piece_id=S72OHKFBDJWBNQ5W5HFPZ2NVGBMVERKOQAE3FNEAYL3CNDENPD3A, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.244885177 (51h56m49.293904019s) [noise=true , address=77.90.31.21:4877 , piece_id=PPRGMYO2DNSDIH3CCE4DKJEKVKZQG6S4AG4QN3B5SF6J67KAGMIA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.252894103 (51h56m49.301912945s) [noise=true , address=45.91.116.13:4155 , piece_id=ITXRUUXIZYQE3EIHOZ2YP4FAGUD2IYTSJLMWDL7XYK2J6JPHMH6Q, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.306583996 (51h56m49.355602842s) [noise=true , address=77.90.31.18:4805 , piece_id=4T72ITVUYQHKAF6F25SOF6UV7JO4Z5W6VLPBTE3U3BH5XNQMEQTA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:59.093756455 (51h57m18.142775311s) [noise=true , address=104.36.82.5:4838 , piece_id=ACWYTC5NJWJTXLKVS4XF6HHAZBQ5FEIP6COOTEEZPCLJDA6CP7XQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 139.59.144.22:53926->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.093890968 (51h57m18.142909825s) [noise=true , address=104.36.82.5:4840 , piece_id=SIYOT6HC7KBXBR52G7CEB75QDYQVJMHUQZRQE4E7XX4XFKH3ETBA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 139.59.144.22:59830->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094001476 (51h57m18.143020332s) [noise=true , address=104.36.82.3:4765 , piece_id=X5IVSDQBVCYXODA3SNVMW27WY6MLUZPOA2RQEIHRFLOPZO5DVANQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 139.59.144.22:48708->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094136882 (51h57m18.143155742s) [noise=true , address=104.36.82.8:4970 , piece_id=RZLKNKN2PLZP44SK4XQ66IXYFK54UX4U3FQZG7BFXGDTUREKZYOA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 139.59.144.22:35836->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094215195 (51h57m18.143234056s) [noise=true , address=104.36.82.11:4114 , piece_id=LHYBE5QL6DJLSZNA47YGOF2ZQ6CSC4HRGHEG2GLQGLLMWEIHF3CQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:51736->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094257493 (51h57m18.143276349s) [noise=true , address=104.36.82.2:4729 , piece_id=AYCGSBJNIVV6ZHUYTIXGHMFF5S47JSA54K26UR25JTHCLTFJCEWA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 139.59.144.22:58390->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094289990 (51h57m18.143308857s) [seg_pos={0 0}, upload=49927]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:53:59.198996113 (51h57m18.248014973s) [upload=49927]: Done uploading segments... Aug 1 21:53:59.199031151 (51h57m18.248050011s) [upload=49927]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.295937187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...X..............X...........................XX] duration: 30.112741913s FAIL (2025-08-01 21:54:30 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 4) Duration: 30.112764434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........................................X.....] duration: 1.282998127s FAIL (2025-08-01 21:54:15 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.283027372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.113351225s FAIL (2025-08-01 21:54:14 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== .879643002 (51h56m33.928661869s) [noise=true , address=104.36.82.47:4552 , piece_id=NS52XP7ZJEHEW5523I3TJ2GAR7QWMX74IJMUJM6UVBWQJOVAQ34A, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.879652397 (51h56m33.928671267s) [noise=true , address=104.36.82.55:4198 , piece_id=OHUUPSXJV3DCQ4KSZBOJAVVN4OWX727EG544HPSKC7O42SK463AQ, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.880052712 (51h56m33.929071577s) [noise=true , address=104.36.82.4:4813 , piece_id=IXPVMHYFYL4OV5GAGU2KAM3XUBYF26KSRORHCTJNZLRFEK4LCR5A, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.880129142 (51h56m33.929147986s) [noise=true , address=104.36.82.53:4124 , piece_id=DEML747ZF4DPNXM7BTS37BLF5W4ITQXHPPUCEKY5FWKHPIB43YFA, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.880506972 (51h56m33.929525838s) [noise=true , address=104.36.82.14:4234 , piece_id=CAKY6X73F6PURIFDLXUQEHS2SMWOSMRGOSHHXWBXVH7JLKGMDLWA, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.880929630 (51h56m33.929948496s) [noise=true , address=64.49.13.140:4513 , piece_id=N55CUZPIOTGERGZHXGPYUF73BSEXMYZGOAV6XETYW7FWESEA2G7A, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.881933565 (51h56m33.930952424s) [noise=true , address=104.36.82.49:4985 , piece_id=ETMOYJK3V6TTSP2IB6N7ZLURJJR5FJKNLFVPHJQCMZV4EQEEH5JQ, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.882634895 (51h56m33.93165377s) [noise=true , address=104.36.82.15:4277 , piece_id=77VNFW2KR3OLCABKUF4RWQRXBOTG5PKZCFEWHIKTFONUGTYGG4HA, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.883292340 (51h56m33.932311206s) [noise=true , address=104.36.82.56:4253 , piece_id=PD4GDHYG7MRCB5CSQJHKXBCQ3GERPF6EQDOI3UHVOWQ4KCJZY4MA, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.883816379 (51h56m33.932835247s) [noise=true , address=45.91.116.13:4167 , piece_id=K4AAH52UHWIXDSMXB32W55AH3V24TTQ5UMKBSCN2SIO4AEG43USA, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:14.886259964 (51h56m33.935278834s) [noise=true , address=104.36.82.42:4818 , piece_id=VCUHK6RNIF7ZKM6YIDQIFGGNUKKQ43W2Q2IOPZN7O3R3INN2VADA, seg_pos={0 0}, upload=49923]: Uploading piece... Aug 1 21:53:15.458787746 (51h56m34.507806602s) [upload=49923]: Next returned nil segment Aug 1 21:53:15.458802512 (51h56m34.507821354s) [upload=49923]: Waiting for error group managing segments... Aug 1 21:53:15.615643410 (51h56m34.66466226s) [noise=true , address=77.90.31.17:4800 , piece_id=GKYQKDISAEWU6EH5NERHNUCGBEFJ2IOSPAGCPAUAOLZGBCMEAZ2A, seg_pos={0 0}, upload=49923]: Done uploading piece. err: <nil> Aug 1 21:53:15.631807888 (51h56m34.680826738s) [noise=true , address=45.91.116.13:4167 , piece_id=K4AAH52UHWIXDSMXB32W55AH3V24TTQ5UMKBSCN2SIO4AEG43USA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: <nil> Aug 1 21:53:16.145587050 (51h56m35.194605893s) [noise=true , address=64.49.13.140:4513 , piece_id=N55CUZPIOTGERGZHXGPYUF73BSEXMYZGOAV6XETYW7FWESEA2G7A, seg_pos={0 0}, upload=49923]: Done uploading piece. err: <nil> Aug 1 21:53:16.245822639 (51h56m35.294841488s) [noise=true , address=79.127.205.245:4230 , piece_id=XXSJDSLNODY4H7KITC32FM72Z2LLSWRR732B5YHOMLPMNN4DGDCA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: <nil> Aug 1 21:54:14.372312446 (51h57m33.421331301s) [noise=true , address=104.36.82.49:4985 , piece_id=ETMOYJK3V6TTSP2IB6N7ZLURJJR5FJKNLFVPHJQCMZV4EQEEH5JQ, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372381458 (51h57m33.421400314s) [noise=true , address=104.36.82.42:4818 , piece_id=VCUHK6RNIF7ZKM6YIDQIFGGNUKKQ43W2Q2IOPZN7O3R3INN2VADA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:104.36.82.42:4818): protocol: rpc: dial tcp 104.36.82.42:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372435140 (51h57m33.421454s) [noise=true , address=104.36.82.53:4124 , piece_id=DEML747ZF4DPNXM7BTS37BLF5W4ITQXHPPUCEKY5FWKHPIB43YFA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: rpc: dial tcp 104.36.82.53:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372484432 (51h57m33.421503288s) [noise=true , address=104.36.82.62:4413 , piece_id=7IXRZ3XXOXGJG45QH6ZRG237BFUPW6FNPEOTRP2BCU264XSE7KSQ, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:1oMVt1KZoNSDX5jKHBXw4WSr69eQYxcY3Z1nxLBsHr19Btw7Gy, address:104.36.82.62:4413): protocol: rpc: dial tcp 104.36.82.62:4413: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372499431 (51h57m33.421518287s) [noise=true , address=104.36.82.14:4234 , piece_id=CAKY6X73F6PURIFDLXUQEHS2SMWOSMRGOSHHXWBXVH7JLKGMDLWA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:12DtCsEnAhXZnGRXjnhNfqNz4RvPnvTMDz1ur7DyZiQtQ7nbwDS, address:104.36.82.14:4234): protocol: rpc: dial tcp 104.36.82.14:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372545588 (51h57m33.421564445s) [noise=true , address=104.36.82.60:4339 , piece_id=3A2M6F4Q3TTM644OOG46R2DYSFXWCZEZLY7OYZPT3OZPK4DC7LOQ, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372555978 (51h57m33.421574833s) [noise=true , address=104.36.82.15:4277 , piece_id=77VNFW2KR3OLCABKUF4RWQRXBOTG5PKZCFEWHIKTFONUGTYGG4HA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: rpc: dial tcp 104.36.82.15:4277: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372595041 (51h57m33.421613897s) [noise=true , address=104.36.82.55:4198 , piece_id=OHUUPSXJV3DCQ4KSZBOJAVVN4OWX727EG544HPSKC7O42SK463AQ, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 139.59.144.22:39686->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372605274 (51h57m33.421624122s) [noise=true , address=104.36.82.56:4253 , piece_id=PD4GDHYG7MRCB5CSQJHKXBCQ3GERPF6EQDOI3UHVOWQ4KCJZY4MA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: rpc: dial tcp 104.36.82.56:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372641598 (51h57m33.421660455s) [noise=true , address=104.36.82.58:4282 , piece_id=LNO4G3GKJQXMZAJQ4HKT4RXCISVQVOIEJAYHIHBBOX5Q3HNAHQ2Q, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:1TcjEH1aW5iW3t2LmMMbn6CCjR5W9cyVDhALoTTmoPtMCFNB5G, address:104.36.82.58:4282): protocol: rpc: dial tcp 104.36.82.58:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372665268 (51h57m33.421684125s) [noise=true , address=104.36.82.47:4552 , piece_id=NS52XP7ZJEHEW5523I3TJ2GAR7QWMX74IJMUJM6UVBWQJOVAQ34A, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372681361 (51h57m33.421700218s) [noise=true , address=104.36.82.53:4152 , piece_id=U6EVG4AARGM44UWNJAKUBWOTUM6RYKU3TT3NLXSV3PKNONR6JECQ, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 139.59.144.22:51192->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372696566 (51h57m33.421715414s) [noise=true , address=104.36.82.10:4104 , piece_id=KSIXNXN36Y5IGPCQ6ICNLEVSCV65GFPM3YPSPJYPG5JMC6IMJCJA, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372714396 (51h57m33.421733252s) [noise=true , address=104.36.82.4:4813 , piece_id=IXPVMHYFYL4OV5GAGU2KAM3XUBYF26KSRORHCTJNZLRFEK4LCR5A, seg_pos={0 0}, upload=49923]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:104.36.82.4:4813): protocol: write tcp 139.59.144.22:51590->104.36.82.4:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:14.372739168 (51h57m33.421758024s) [seg_pos={0 0}, upload=49923]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Aug 1 21:54:14.482822146 (51h57m33.531840993s) [upload=49923]: Done uploading segments... Aug 1 21:54:14.482845615 (51h57m33.531864472s) [upload=49923]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.113375214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001261919s FAIL (2025-08-01 21:54:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001283851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.107946852s FAIL (2025-08-01 21:53:59 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 21:52:17.921392832 (51h55m36.970411696s) [upload=49907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:52:18.055386824 (51h55m37.104405689s) [upload=49907]: Done uploading segments... Aug 1 21:52:18.055422517 (51h55m37.104441366s) [upload=49907]: Upload finished. err: <nil> Aug 1 21:53:29.365104050 (51h56m48.414122906s) [upload=49927]: Starting upload Aug 1 21:53:29.365175879 (51h56m48.414194743s) [upload=49927]: Done starting upload Aug 1 21:53:29.365455201 (51h56m48.414474043s) [upload=49927]: Uploading segments... Aug 1 21:53:29.365486823 (51h56m48.414505686s) [upload=49927]: Got next segment. Inline: false Aug 1 21:53:29.365542702 (51h56m48.414561545s) [upload=49927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:53:29.365546365 (51h56m48.414565221s) [upload=49927]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:53:29.472662631 (51h56m48.52168149s) [seg_pos={0 0}, upload=49927]: Begin upload segment... Aug 1 21:53:29.472724832 (51h56m48.521743682s) [seg_pos={0 0}, upload=49927]: Done begin upload segment. Aug 1 21:53:29.472774947 (51h56m48.521793806s) [noise=true , address=45.91.116.8:4047 , piece_id=N3R4B6GHWNAX45LK6WJN5PKY3ISWXG3SEBM4Z2TOKISG3X4RWRUQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.472782672 (51h56m48.521801528s) [noise=true , address=104.36.82.5:4838 , piece_id=ACWYTC5NJWJTXLKVS4XF6HHAZBQ5FEIP6COOTEEZPCLJDA6CP7XQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473218746 (51h56m48.522237602s) [noise=true , address=45.91.116.10:4081 , piece_id=H25Q5MVZIXIILL7X6HXNFDKOSZVLXC4RML7YZJ4JTNCCKZUILM4A, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473262022 (51h56m48.52228088s) [noise=true , address=77.90.31.18:4805 , piece_id=4T72ITVUYQHKAF6F25SOF6UV7JO4Z5W6VLPBTE3U3BH5XNQMEQTA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473648227 (51h56m48.522667086s) [noise=true , address=45.91.116.6:4975 , piece_id=S72OHKFBDJWBNQ5W5HFPZ2NVGBMVERKOQAE3FNEAYL3CNDENPD3A, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.473800893 (51h56m48.522819766s) [noise=true , address=104.36.82.5:4840 , piece_id=SIYOT6HC7KBXBR52G7CEB75QDYQVJMHUQZRQE4E7XX4XFKH3ETBA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474079503 (51h56m48.523098359s) [noise=true , address=77.90.31.21:4877 , piece_id=PPRGMYO2DNSDIH3CCE4DKJEKVKZQG6S4AG4QN3B5SF6J67KAGMIA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474256677 (51h56m48.523275534s) [noise=true , address=77.90.31.16:4733 , piece_id=JXEWLMPA3TTXHJROG2QAH7VFYLDO5WBAYJSHTG4ANSS3SL66BZ6A, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474428248 (51h56m48.523447107s) [noise=true , address=45.91.116.7:4010 , piece_id=QNAUYQ4R7KGAOPR63GSBWLU47ORBKUFQOL4D4WWS32KS3VSD4YAQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.474718468 (51h56m48.523737324s) [noise=true , address=45.91.116.10:4082 , piece_id=DM2AG74TCYCXLPZDK2TDZ2QIBWMCWJGVSDEMS2ZV47SO4ZWSFZKQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.475166790 (51h56m48.524185646s) [noise=true , address=45.91.116.13:4155 , piece_id=ITXRUUXIZYQE3EIHOZ2YP4FAGUD2IYTSJLMWDL7XYK2J6JPHMH6Q, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.478494979 (51h56m48.527513844s) [noise=true , address=104.36.82.2:4729 , piece_id=AYCGSBJNIVV6ZHUYTIXGHMFF5S47JSA54K26UR25JTHCLTFJCEWA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.479015210 (51h56m48.528034066s) [noise=true , address=104.36.82.11:4114 , piece_id=LHYBE5QL6DJLSZNA47YGOF2ZQ6CSC4HRGHEG2GLQGLLMWEIHF3CQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.479563863 (51h56m48.528582735s) [noise=true , address=104.36.82.3:4765 , piece_id=X5IVSDQBVCYXODA3SNVMW27WY6MLUZPOA2RQEIHRFLOPZO5DVANQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.480035123 (51h56m48.529053986s) [noise=true , address=104.36.82.8:4970 , piece_id=RZLKNKN2PLZP44SK4XQ66IXYFK54UX4U3FQZG7BFXGDTUREKZYOA, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:29.480380992 (51h56m48.529399835s) [noise=true , address=77.90.31.14:4662 , piece_id=7I44ITGD3VFOQYFY23XUUYJ3OA3DX3W77A3X5VUKF2HTJR7U7KQQ, seg_pos={0 0}, upload=49927]: Uploading piece... Aug 1 21:53:30.090118150 (51h56m49.139137016s) [upload=49927]: Next returned nil segment Aug 1 21:53:30.090139229 (51h56m49.139158095s) [upload=49927]: Waiting for error group managing segments... Aug 1 21:53:30.231512679 (51h56m49.280531522s) [noise=true , address=77.90.31.16:4733 , piece_id=JXEWLMPA3TTXHJROG2QAH7VFYLDO5WBAYJSHTG4ANSS3SL66BZ6A, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.233188516 (51h56m49.282207364s) [noise=true , address=45.91.116.10:4082 , piece_id=DM2AG74TCYCXLPZDK2TDZ2QIBWMCWJGVSDEMS2ZV47SO4ZWSFZKQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.234293524 (51h56m49.283312395s) [noise=true , address=77.90.31.14:4662 , piece_id=7I44ITGD3VFOQYFY23XUUYJ3OA3DX3W77A3X5VUKF2HTJR7U7KQQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.236894742 (51h56m49.285913585s) [noise=true , address=45.91.116.7:4010 , piece_id=QNAUYQ4R7KGAOPR63GSBWLU47ORBKUFQOL4D4WWS32KS3VSD4YAQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.239401166 (51h56m49.28842002s) [noise=true , address=45.91.116.8:4047 , piece_id=N3R4B6GHWNAX45LK6WJN5PKY3ISWXG3SEBM4Z2TOKISG3X4RWRUQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.239827716 (51h56m49.288846564s) [noise=true , address=45.91.116.10:4081 , piece_id=H25Q5MVZIXIILL7X6HXNFDKOSZVLXC4RML7YZJ4JTNCCKZUILM4A, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.243422005 (51h56m49.292440857s) [noise=true , address=45.91.116.6:4975 , piece_id=S72OHKFBDJWBNQ5W5HFPZ2NVGBMVERKOQAE3FNEAYL3CNDENPD3A, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.244885177 (51h56m49.293904019s) [noise=true , address=77.90.31.21:4877 , piece_id=PPRGMYO2DNSDIH3CCE4DKJEKVKZQG6S4AG4QN3B5SF6J67KAGMIA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.252894103 (51h56m49.301912945s) [noise=true , address=45.91.116.13:4155 , piece_id=ITXRUUXIZYQE3EIHOZ2YP4FAGUD2IYTSJLMWDL7XYK2J6JPHMH6Q, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:30.306583996 (51h56m49.355602842s) [noise=true , address=77.90.31.18:4805 , piece_id=4T72ITVUYQHKAF6F25SOF6UV7JO4Z5W6VLPBTE3U3BH5XNQMEQTA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: <nil> Aug 1 21:53:59.093756455 (51h57m18.142775311s) [noise=true , address=104.36.82.5:4838 , piece_id=ACWYTC5NJWJTXLKVS4XF6HHAZBQ5FEIP6COOTEEZPCLJDA6CP7XQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 139.59.144.22:53926->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.093890968 (51h57m18.142909825s) [noise=true , address=104.36.82.5:4840 , piece_id=SIYOT6HC7KBXBR52G7CEB75QDYQVJMHUQZRQE4E7XX4XFKH3ETBA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 139.59.144.22:59830->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094001476 (51h57m18.143020332s) [noise=true , address=104.36.82.3:4765 , piece_id=X5IVSDQBVCYXODA3SNVMW27WY6MLUZPOA2RQEIHRFLOPZO5DVANQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 139.59.144.22:48708->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094136882 (51h57m18.143155742s) [noise=true , address=104.36.82.8:4970 , piece_id=RZLKNKN2PLZP44SK4XQ66IXYFK54UX4U3FQZG7BFXGDTUREKZYOA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 139.59.144.22:35836->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094215195 (51h57m18.143234056s) [noise=true , address=104.36.82.11:4114 , piece_id=LHYBE5QL6DJLSZNA47YGOF2ZQ6CSC4HRGHEG2GLQGLLMWEIHF3CQ, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:51736->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094257493 (51h57m18.143276349s) [noise=true , address=104.36.82.2:4729 , piece_id=AYCGSBJNIVV6ZHUYTIXGHMFF5S47JSA54K26UR25JTHCLTFJCEWA, seg_pos={0 0}, upload=49927]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 139.59.144.22:58390->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:59.094289990 (51h57m18.143308857s) [seg_pos={0 0}, upload=49927]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:53:59.198996113 (51h57m18.248014973s) [upload=49927]: Done uploading segments... Aug 1 21:53:59.199031151 (51h57m18.248050011s) [upload=49927]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.107967636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........................................X....] duration: 1.060323064s FAIL (2025-08-01 21:53:45 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.060389145s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)