Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 2.563801251s FAIL (2025-08-01 22:20:00 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.563833049s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.030169237s FAIL (2025-08-01 22:19:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.030190991s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....................X................X.X...X...............] duration: 1.193482735s FAIL (2025-08-01 22:19:42 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.193508642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 2.81051147s FAIL (2025-08-01 22:19:37 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.810530372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 3.54259819s FAIL (2025-08-01 22:19:36 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.542623557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XXXXX...........X....................] duration: 2.62502085s FAIL (2025-08-01 22:19:36 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 2.625102376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.672006556s FAIL (2025-08-01 22:19:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.672027705s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 2.344959507s FAIL (2025-08-01 22:19:14 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.34498787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XXXXX...........X...................] duration: 3.110814478s FAIL (2025-08-01 22:19:12 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.110846261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 3.404951912s FAIL (2025-08-01 22:19:05 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.404983393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration: 3.253752015s FAIL (2025-08-01 22:18:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.253779419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................X................X.X...X..............] duration: 1.810056329s FAIL (2025-08-01 22:18:50 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.810078743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 3.249432675s FAIL (2025-08-01 22:18:45 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.249454794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 2.598895123s FAIL (2025-08-01 22:18:41 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.598922316s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.608153685s FAIL (2025-08-01 22:18:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.608193972s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 2.335066093s FAIL (2025-08-01 22:18:17 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.335090586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XXXXX...........X..................] duration: 2.91397932s FAIL (2025-08-01 22:18:15 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 2.914007206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................X................X.X...X.............] duration: 986.281051ms FAIL (2025-08-01 22:18:10 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 986.312618ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration: 3.040591368s FAIL (2025-08-01 22:18:06 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.040621316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 2.313816391s FAIL (2025-08-01 22:17:54 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.313840979s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration: 4.540307523s FAIL (2025-08-01 22:17:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.54040235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.822677802s FAIL (2025-08-01 22:17:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.822722497s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration: 2.852752017s FAIL (2025-08-01 22:17:26 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.85278594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XXXXX...........X.................] duration: 3.525950353s FAIL (2025-08-01 22:17:05 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.525982099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.665867319s FAIL (2025-08-01 22:17:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.66590021s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 2.475818621s FAIL (2025-08-01 22:17:00 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.475857225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................X................X.X...X............] duration: 1.019275191s FAIL (2025-08-01 22:16:57 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.019303914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration: 5.138431352s FAIL (2025-08-01 22:16:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.138788418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration: 3.124527975s FAIL (2025-08-01 22:16:28 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.124615018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.738283216s FAIL (2025-08-01 22:16:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.738309123s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration: 13.940758225s FAIL (2025-08-01 22:16:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.940784791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XXXXX...........X................] duration: 3.808102498s FAIL (2025-08-01 22:16:08 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.808129846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................XXXX............................] duration: 3.174259506s FAIL (2025-08-01 22:16:04 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.17429631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.907592001s FAIL (2025-08-01 22:16:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.90761407s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 2.620059708s FAIL (2025-08-01 22:16:01 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.620087957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........................X................X.X...X...........] duration: 869.482456ms FAIL (2025-08-01 22:15:44 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 869.524158ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...........................XXXXX...........X...............] duration: 3.80686236s FAIL (2025-08-01 22:15:42 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.806893809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 3.250051026s FAIL (2025-08-01 22:15:41 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.250073828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........................X................X.X...X..........] duration: 2.214298714s FAIL (2025-08-01 22:15:15 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.214354967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...........................XXXXX...........X..............] duration: 3.599851055s FAIL (2025-08-01 22:15:10 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.599936724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 2.328656302s FAIL (2025-08-01 22:14:54 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.328682763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration: 3.554127734s FAIL (2025-08-01 22:14:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554150232s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.904569963s FAIL (2025-08-01 22:14:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.904750256s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration: 2.68918124s FAIL (2025-08-01 22:14:42 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.689199247s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration: 21.497559217s FAIL (2025-08-01 22:14:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.497587336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........................X................X.X...X.........] duration: 1.248612534s FAIL (2025-08-01 22:14:25 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.248641292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...........................XXXXX...........X.............] duration: 3.015574643s FAIL (2025-08-01 22:14:15 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.0156043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration: 3.027911997s FAIL (2025-08-01 22:14:14 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.027949286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.855068112s FAIL (2025-08-01 22:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.855094109s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................................XXXX........................] duration: 4.356214563s FAIL (2025-08-01 22:13:55 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 4.356237705s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)