Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...........................XXXXX...........X............] duration: 3.756456491s FAIL (2025-08-01 22:13:47 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.756489546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 2.687372531s FAIL (2025-08-01 22:13:45 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.687405153s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................................XXXXX.......................] duration: 14.943316639s FAIL (2025-08-01 22:13:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.943336983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [............................X................X.X...X........] duration: 1.29012379s FAIL (2025-08-01 22:13:30 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.290147334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.831739316s FAIL (2025-08-01 22:13:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.831759823s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X...........................XXXXX...........X...........] duration: 2.586363805s FAIL (2025-08-01 22:13:23 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 2.586391223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 2.654566499s FAIL (2025-08-01 22:13:12 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.65459759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................................XXXXX......................] duration: 7.868617051s FAIL (2025-08-01 22:12:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.86863335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X...........................XXXXX...........X..........] duration: 3.587037028s FAIL (2025-08-01 22:12:56 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.587064403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.004106575s FAIL (2025-08-01 22:12:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.004368608s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 3.987156817s FAIL (2025-08-01 22:12:40 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.987185913s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................................XXXXX.....................] duration: 23.302156446s FAIL (2025-08-01 22:12:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.302180091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 2.445965736s FAIL (2025-08-01 22:12:31 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.445991767s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.74604007s FAIL (2025-08-01 22:12:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.746085948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............................X................X.X...X.......] duration: 1.056827828s FAIL (2025-08-01 22:12:17 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.056949041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................................XXXXX....................] duration: 18.375352544s FAIL (2025-08-01 22:12:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.375377008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X...........................XXXXX...........X.........] duration: 4.074882778s FAIL (2025-08-01 22:12:03 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 4.07491165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............................X................X.X...X......] duration: 1.980875675s FAIL (2025-08-01 22:11:58 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.980905526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 2.400800376s FAIL (2025-08-01 22:11:39 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.400826509s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.814396356s FAIL (2025-08-01 22:11:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.814416608s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration: 17.913297662s FAIL (2025-08-01 22:11:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.913319154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 3.456100033s FAIL (2025-08-01 22:11:38 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.456127403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.601695211s FAIL (2025-08-01 22:11:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.601717697s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...............................X................X.X...X.....] duration: 2.037537567s FAIL (2025-08-01 22:11:11 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.037563665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X...........................XXXXX...........X........] duration: 3.525626536s FAIL (2025-08-01 22:11:06 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.525653974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 3.606672936s FAIL (2025-08-01 22:10:53 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.606719905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.867445672s FAIL (2025-08-01 22:10:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.867458992s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 2.607180994s FAIL (2025-08-01 22:10:51 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.607210292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration: 12.252049017s FAIL (2025-08-01 22:10:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.25206184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 2.489177291s FAIL (2025-08-01 22:10:28 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.48920524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 3.446018407s FAIL (2025-08-01 22:10:25 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.446042652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [................................X................X.X...X....] duration: 1.303700043s FAIL (2025-08-01 22:10:24 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.303725239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration: 12.205645361s FAIL (2025-08-01 22:10:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.205666722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X...........................XXXXX...........X.......] duration: 5.583943529s FAIL (2025-08-01 22:10:13 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 5.583976215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.860168636s FAIL (2025-08-01 22:09:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.860192945s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X...........................XXXXX...........X......] duration: 3.288790867s FAIL (2025-08-01 22:09:51 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.288819824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 3.162992117s FAIL (2025-08-01 22:09:43 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.163025042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.................................X................X.X...X...] duration: 1.086505234s FAIL (2025-08-01 22:09:43 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.086529216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 2.595218955s FAIL (2025-08-01 22:09:40 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.595244798s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration: 3.138329229s FAIL (2025-08-01 22:09:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.138353501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X...........................XXXXX...........X.....] duration: 4.965150848s FAIL (2025-08-01 22:09:15 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 4.965180973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration: 9.048239429s FAIL (2025-08-01 22:09:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.048262189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..................................X................X.X...X..] duration: 4.708396146s FAIL (2025-08-01 22:09:07 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 4.708419381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 2.537637253s FAIL (2025-08-01 22:08:50 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.537659786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X...........................XXXXX...........X....] duration: 5.619885033s FAIL (2025-08-01 22:08:49 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 5.619974105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 3.313132044s FAIL (2025-08-01 22:08:48 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.313164648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...................................X................X.X...X.] duration: 867.932555ms FAIL (2025-08-01 22:08:43 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 867.959323ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.654118243s FAIL (2025-08-01 22:08:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.654142535s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................................X................X.X...X] duration: 2.901433349s FAIL (2025-08-01 22:08:21 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.901471946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 4.049199557s FAIL (2025-08-01 22:08:12 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 4.049221729s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)