Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration: 3.060335522s FAIL (2025-08-01 22:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.060361455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.992004036s FAIL (2025-08-01 22:07:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.992032813s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X...........................XXXXX...........X..] duration: 3.251695716s FAIL (2025-08-01 22:07:22 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 3.251728987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 2.748269159s FAIL (2025-08-01 22:07:13 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.748294501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration: 6.937520212s FAIL (2025-08-01 22:07:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.937540555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................................X................X.X..] duration: 3.058022707s FAIL (2025-08-01 22:07:05 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 3.058141401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X...........................XXXXX...........X.] duration: 10.540701324s FAIL (2025-08-01 22:06:59 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 10.540727743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 2.580408127s FAIL (2025-08-01 22:06:54 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.580428811s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.805293167s FAIL (2025-08-01 22:06:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.805320459s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 3.496857703s FAIL (2025-08-01 22:06:42 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.496880412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration: 7.846032664s FAIL (2025-08-01 22:06:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.846052631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..............X...........................XXXXX...........X] duration: 9.915797955s FAIL (2025-08-01 22:06:29 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 9.915836173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X................X.X.] duration: 2.835371771s FAIL (2025-08-01 22:06:29 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.835394603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 2.651572521s FAIL (2025-08-01 22:06:16 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.651600514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 2.661241994s FAIL (2025-08-01 22:06:01 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.66126828s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.768321625s FAIL (2025-08-01 22:05:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.76834756s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration: 3.22411946s FAIL (2025-08-01 22:05:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.224141967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 3.481151061s FAIL (2025-08-01 22:05:27 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.481178733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................X................X.X] duration: 3.752973171s FAIL (2025-08-01 22:05:22 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 3.752997089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..............X...........................XXXXX...........] duration: 4.941352629s FAIL (2025-08-01 22:05:20 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 4.941379828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXXXX.........] duration: 12.543448479s FAIL (2025-08-01 22:05:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.543469438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X................X.] duration: 871.087889ms FAIL (2025-08-01 22:05:01 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 871.109516ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 3.092789537s FAIL (2025-08-01 22:04:59 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.092818571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 2.635815576s FAIL (2025-08-01 22:04:51 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.635844992s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.719764736s FAIL (2025-08-01 22:04:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.71978768s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 3.083415888s FAIL (2025-08-01 22:04:34 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.083443024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..............X...........................XXXXX..........] duration: 3.206539303s FAIL (2025-08-01 22:04:29 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 3.206729496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXXXX........] duration: 12.237744463s FAIL (2025-08-01 22:04:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.237770187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 2.656290614s FAIL (2025-08-01 22:04:24 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.656316891s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.679869501s FAIL (2025-08-01 22:04:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.680089198s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X................X] duration: 10.247187405s FAIL (2025-08-01 22:04:07 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 10.247226456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 2.479522569s FAIL (2025-08-01 22:03:47 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.479567074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 3.57822673s FAIL (2025-08-01 22:03:43 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.578263821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..............X...........................XXXXX.........] duration: 4.449076346s FAIL (2025-08-01 22:03:39 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 4.449107069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.622561579s FAIL (2025-08-01 22:03:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.622581543s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................................................XXXXX.......] duration: 18.27857434s FAIL (2025-08-01 22:03:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.278623393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 21.638820356s FAIL (2025-08-01 22:03:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.638844947s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 11.655810054s FAIL (2025-08-01 22:03:16 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 11.655837276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 2.939645175s FAIL (2025-08-01 22:03:09 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 2.939676894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...X..............X...........................XXXXX........] duration: 5.321810172s FAIL (2025-08-01 22:03:09 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 5.32185146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 2.551915126s FAIL (2025-08-01 22:02:38 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .22:53596->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:33.055690917 (51h57m52.104709781s) [seg_pos={0 0}, upload=49945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:54:33.055805249 (51h57m52.104824107s) [upload=49945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:54:33.055809880 (51h57m52.104828741s) [upload=49945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:54:33.194928120 (51h57m52.243946977s) [upload=49945]: Done uploading segments... Aug 1 21:54:33.194945396 (51h57m52.243964253s) [upload=49945]: Upload finished. err: <nil> Aug 1 21:55:46.230084799 (51h59m5.279103645s) [upload=49961]: Starting upload Aug 1 21:55:46.230157514 (51h59m5.279176371s) [upload=49961]: Done starting upload Aug 1 21:55:46.230274904 (51h59m5.27929376s) [upload=49961]: Uploading segments... Aug 1 21:55:46.230560643 (51h59m5.2795795s) [upload=49961]: Got next segment. Inline: false Aug 1 21:55:46.230599659 (51h59m5.279618515s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:55:46.230604686 (51h59m5.279623541s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:55:46.346974027 (51h59m5.395992884s) [seg_pos={0 0}, upload=49961]: Begin upload segment... Aug 1 21:55:46.347038349 (51h59m5.396057218s) [seg_pos={0 0}, upload=49961]: Done begin upload segment. Aug 1 21:55:46.347046053 (51h59m5.39606492s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347057696 (51h59m5.39607656s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.347629792 (51h59m5.39664866s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348574718 (51h59m5.397593574s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.348953096 (51h59m5.397971953s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349271454 (51h59m5.39829031s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349352370 (51h59m5.398371218s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.349882107 (51h59m5.398900964s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350104927 (51h59m5.399123784s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350790749 (51h59m5.399809608s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.350863437 (51h59m5.399882283s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.351548122 (51h59m5.400566981s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352004051 (51h59m5.40102292s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.352517073 (51h59m5.401535939s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353268671 (51h59m5.402287527s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.353977370 (51h59m5.402996231s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Uploading piece... Aug 1 21:55:46.968080869 (51h59m6.017099724s) [upload=49961]: Next returned nil segment Aug 1 21:55:46.968133606 (51h59m6.017152463s) [upload=49961]: Waiting for error group managing segments... Aug 1 21:55:47.098589247 (51h59m6.147608096s) [noise=true , address=45.91.116.13:4154 , piece_id=WHHLEQRXRBXAJ367JXKWI43JUQBE5ZX6Y7XXOT2IF6ZBDIHIS7JA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.100044573 (51h59m6.149063423s) [noise=true , address=45.91.116.5:4297 , piece_id=NHCRTZ6R5CSNYLVLCD2DIC7HQFTSVSJFT4ISGD3XG6JW6JVQJHBA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.101668289 (51h59m6.150687131s) [noise=true , address=77.90.31.13:4626 , piece_id=STWONIC3TS6CPK3T4T3RDZ4P2NAZP7PJI6I3JVTVYXK2ZTQN2SWQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102419178 (51h59m6.151438036s) [noise=true , address=77.90.31.11:4554 , piece_id=SDNLMCMTPH4WSANKX3QMPH3WGUF5MWOW4PCEWRMV4UDZZR3LXRAQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.102866594 (51h59m6.15188545s) [noise=true , address=77.90.31.15:4699 , piece_id=LW5RSRFINNVBBBQFVZMAXIPC23LVXM6FH3ZEJLXIQGQ3G3N3XDYA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.113602204 (51h59m6.162621057s) [noise=true , address=77.90.31.21:4879 , piece_id=J2OX4MNZW4NKFZ6B2POOA4X2H7M34ECXUOTNY64NCHJ466M55ODA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.116304862 (51h59m6.165323707s) [noise=true , address=45.91.116.5:4299 , piece_id=IRLU5RRBWUM4P6AYNDSBXPSNR4T5RYRVBNFUUCTDZKVSXWHAFJRA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.176733563 (51h59m6.225752414s) [noise=true , address=45.91.116.10:4082 , piece_id=22NC6GOPCRI7SM2IXAL3H5XK5JCFVMP27BUHYVTFLR4PZRYIVFHA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.178397925 (51h59m6.227416782s) [noise=true , address=45.91.116.11:4118 , piece_id=B2W24UMGL3O57L7JH5F2YCI6GWZLV722RETKGO3J6JITPQ7ILQEA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:55:47.194101480 (51h59m6.243120338s) [noise=true , address=45.91.116.4:4939 , piece_id=VXC7SNVGRXL5WCHW34EQ5GKQUFA64PIBQKCUEF7MPPJ6QAJLX7PA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: <nil> Aug 1 21:56:15.949939028 (51h59m34.998957884s) [noise=true , address=104.36.82.14:4222 , piece_id=O5FMFQVXPZZX3P7FOFLUZPCM2EG64PHBPISWJCEFVB4ZPBMJFTJA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950049627 (51h59m34.999068482s) [noise=true , address=104.36.82.4:4801 , piece_id=I7UR3CSEPFMCJHHYZB5LHXICD4XA72BJJHRZ6PQJWHOU4XRDFR5Q, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:54374->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950108588 (51h59m34.999127447s) [noise=true , address=104.36.82.14:4221 , piece_id=7IEXDMPWVXDFCBMKD64MVMIM74LSA7JZ6OTQ33YP7GYM2EMIWFPQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950121106 (51h59m34.999139961s) [noise=true , address=104.36.82.6:4876 , piece_id=IH6VXI57IFIPOCF545QE7FCH3AXLWSKFUKK6OCS5QRSXZADALCZA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950170357 (51h59m34.999189213s) [noise=true , address=104.36.82.2:4732 , piece_id=AZLVCIR4IPWILLC6WFB3P3BETARFQRRVUNQFYMBAEBFNSF6B5GJQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:15.950193183 (51h59m34.999212039s) [noise=true , address=104.36.82.7:4910 , piece_id=FAJUYGJV5AO7OAEOXQ3LUCCMGV7MTPQRKNIO5ZSU7ZXIYB7IHEEQ, seg_pos={0 0}, upload=49961]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:56:15.950211938 (51h59m34.999230797s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 1 21:56:16.066372604 (51h59m35.115391463s) [upload=49961]: Done uploading segments... Aug 1 21:56:16.066422132 (51h59m35.115440989s) [upload=49961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.551941159s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XXXXX......] duration: 3.607563494s FAIL (2025-08-01 22:02:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607579485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...X..............X...........................XXXXX.......] duration: 9.770728045s FAIL (2025-08-01 22:02:29 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 5) Duration: 9.770755625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 3.798738043s FAIL (2025-08-01 22:02:23 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.798780434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X...........................................X...............] duration: 897.386983ms FAIL (2025-08-01 22:02:16 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 897.402716ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X...........................................X..............] duration: 17.032668495s FAIL (2025-08-01 22:02:15 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 17.032697102s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXXXX.....] duration: 10.688176841s FAIL (2025-08-01 22:02:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.688196154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.093904603s FAIL (2025-08-01 22:02:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.093926918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 3.878055748s FAIL (2025-08-01 22:02:07 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 9m38.069106s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020470612 (51h59m38.069489475s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020624289 (51h59m38.069643151s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.020816317 (51h59m38.069835182s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021032875 (51h59m38.07005173s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021139334 (51h59m38.070158192s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021491715 (51h59m38.070510572s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.021799641 (51h59m38.070818504s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022118253 (51h59m38.071137118s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022409929 (51h59m38.071428788s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022498047 (51h59m38.071516907s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022691602 (51h59m38.071710553s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.022970433 (51h59m38.071989292s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023379341 (51h59m38.072398197s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.023728110 (51h59m38.072746973s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024013606 (51h59m38.073032462s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.024307311 (51h59m38.073326167s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Uploading piece... Aug 1 21:56:19.585306474 (51h59m38.634325336s) [upload=49970]: Next returned nil segment Aug 1 21:56:19.585844250 (51h59m38.634863113s) [upload=49970]: Waiting for error group managing segments... Aug 1 21:56:19.779902959 (51h59m38.828921815s) [noise=true , address=79.127.219.38:4008 , piece_id=WLL35AZAUSB4DKPUNO6Y645FQPGLNOHWP3HZBW3P6QERFNIUI7TQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.782349572 (51h59m38.831368429s) [noise=true , address=79.127.219.51:4183 , piece_id=UL7CBKW5J5RXPXUMXNE4MTNQWT5M4SK2YZQ4LSWXWXT52VSQ6C7Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.811726022 (51h59m38.860744878s) [noise=true , address=77.90.31.14:4683 , piece_id=6S2TDGTP3WO5ZIIXPPTT6SMUQ25IBN27VSCVQLD3VABBJVLLYE2Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:19.955645898 (51h59m39.00466474s) [noise=true , address=79.127.201.214:4296 , piece_id=2QMYUO2REROO5WO5V64AY33V24NFGKL7MCEGZ4S6BKACUAGMQ7WQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.035748219 (51h59m39.084767069s) [noise=true , address=79.127.220.101:4075 , piece_id=QNOQ2OR3BTGGYDTUUTDIQSUOCJQPWLI32PDNQ2O2PFCJJ5XXZCGQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.280949371 (51h59m39.32996822s) [noise=true , address=193.149.164.143:4695 , piece_id=QDQSS72VKNEWG64ZIXFZAHHZNWWD5TW5NIYBUWZGJILVGHY3C62A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.300968244 (51h59m39.349987106s) [noise=true , address=193.149.164.143:4702 , piece_id=Q2CQYLYJB7OKQB6CYDE2AO42DX5WR4Z526TZRE6HNH2UYOP3UP3Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.351942831 (51h59m39.400961677s) [noise=true , address=79.127.205.234:4586 , piece_id=XC74SLX4G5ZTXDMLLLZCKBH66GO7BAAXQZYYMJR322O3ZOYMQRPQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:56:20.394604251 (51h59m39.443623095s) [noise=true , address=79.127.205.246:4267 , piece_id=B6HTPJVVIHRF5DT53INZNECNUN67GCZWC3GVYXMFEX5KIQ6JA7BA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: <nil> Aug 1 21:57:18.556247169 (52h0m37.605266026s) [noise=true , address=104.36.82.62:4414 , piece_id=DTASFRSKQND72TY7LEEBRA5Y6MW2BCIJW3MYIVQYO5XJ2UPV4ZSQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:104.36.82.62:4414): protocol: rpc: dial tcp 104.36.82.62:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556322491 (52h0m37.605341333s) [noise=true , address=104.36.82.11:4142 , piece_id=SKZVYCAMMLMBT7J65ZTBGOQNOGQZCYI66HSPEUXGYIS4HUNA43VA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: rpc: dial tcp 104.36.82.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556360960 (52h0m37.605379802s) [noise=true , address=104.36.82.5:4857 , piece_id=XY2C3HFUQAGO4BUKAZDEKA6HIKLQG5ACJF4JO75RWOK6YB5AJTCQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556397616 (52h0m37.605416458s) [noise=true , address=104.36.82.13:4204 , piece_id=UC37GRXZZJA7BBWJOEXYVA7IBQQUL6I2NMH7L2MOT6GV6PPTX5XQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556434146 (52h0m37.605452988s) [noise=true , address=104.36.82.47:4546 , piece_id=VV4N5YADBSZKM6RIBN2ISPEG3V5YOW3W2PB74RLQV7NUTE6JY3NQ, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556469345 (52h0m37.605488188s) [noise=true , address=104.36.82.52:4106 , piece_id=Q7PTI5APM74OVTKGDZN7RLIDOSFVQATOQBBWO2FVYDPOS5UUHF5A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556525011 (52h0m37.605543853s) [noise=true , address=104.36.82.11:4132 , piece_id=MTFNNEROVLPU5QWJMGXQTBSKQHR43WQUY37LQ7OORYRBE664Z25A, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556557058 (52h0m37.6055759s) [noise=true , address=104.36.82.9:4008 , piece_id=3W56A7UV6H5HHYE6IHN7I373HJERBEYC3CHZBL4SZFQPFRXGNY4Q, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556598391 (52h0m37.605617233s) [noise=true , address=104.36.82.7:4926 , piece_id=XNIY325K22ZRMZJHKLFWKQI7WAK7BAXKMZKZADQCDK52DZKAITZA, seg_pos={0 0}, upload=49970]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: write tcp 139.59.144.22:34412->104.36.82.7:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:18.556619047 (52h0m37.605637903s) [seg_pos={0 0}, upload=49970]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 1 21:57:18.663386709 (52h0m37.712405553s) [upload=49970]: Done uploading segments... Aug 1 21:57:18.663411371 (52h0m37.712430213s) [upload=49970]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 3.878083551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXXXX....] duration: 17.311056138s FAIL (2025-08-01 22:01:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.31127471s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)