Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X.......................XX.X..........X....] duration: 4.063637632s FAIL (2025-08-07 21:45:49 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 3 (12h3m20.890980883s) [upload=12439]: Starting upload Aug 7 21:42:59.146979817 (12h3m20.89103668s) [upload=12439]: Done starting upload Aug 7 21:42:59.147227284 (12h3m20.891284134s) [upload=12439]: Uploading segments... Aug 7 21:42:59.147266319 (12h3m20.891323188s) [upload=12439]: Got next segment. Inline: false Aug 7 21:42:59.147299756 (12h3m20.891356609s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:59.147305046 (12h3m20.891361896s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:59.534612872 (12h3m21.278669723s) [seg_pos={0 0}, upload=12439]: Begin upload segment... Aug 7 21:42:59.534692220 (12h3m21.278749069s) [seg_pos={0 0}, upload=12439]: Done begin upload segment. Aug 7 21:42:59.534758302 (12h3m21.278815173s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535350842 (12h3m21.279407711s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535933152 (12h3m21.279990018s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536298922 (12h3m21.280355769s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536689502 (12h3m21.280746369s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537035690 (12h3m21.281092562s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537373880 (12h3m21.281430743s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537716758 (12h3m21.281773618s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537995761 (12h3m21.282052656s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538051062 (12h3m21.282107912s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538386676 (12h3m21.282443539s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538674756 (12h3m21.282731616s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538946838 (12h3m21.283003698s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539233583 (12h3m21.283290444s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539738096 (12h3m21.28379497s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540187297 (12h3m21.284244165s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540518593 (12h3m21.284575456s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540774182 (12h3m21.284831043s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:43:00.120361086 (12h3m21.864417951s) [upload=12439]: Next returned nil segment Aug 7 21:43:00.120407815 (12h3m21.864464686s) [upload=12439]: Waiting for error group managing segments... Aug 7 21:43:00.271581920 (12h3m22.015638774s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.301379636 (12h3m22.0454365s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.315130147 (12h3m22.05918701s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.351162615 (12h3m22.095219465s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.383164149 (12h3m22.127221015s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.418050578 (12h3m22.162107442s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.451823428 (12h3m22.195880292s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.467192643 (12h3m22.211249504s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.495563998 (12h3m22.239620847s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.580244990 (12h3m22.324301836s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.647231825 (12h3m22.391288672s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.702946639 (12h3m22.447003485s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.748375718 (12h3m22.492432579s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801483011 (12h3m22.545539858s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801509099 (12h3m22.545565964s) [seg_pos={0 0}, upload=12439]: Segment reached optimal threshold of 14 pieces. Aug 7 21:43:00.801678905 (12h3m22.545735756s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802028506 (12h3m22.546085367s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802168197 (12h3m22.546225051s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802240477 (12h3m22.546297327s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: read tcp 139.59.144.22:42920->104.36.82.37:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:43:00.802254494 (12h3m22.546311343s) [seg_pos={0 0}, upload=12439]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:43:00.802379259 (12h3m22.546436112s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:43:00.802386197 (12h3m22.546443069s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.245820404 (12h3m23.98987728s) [upload=12439]: Done uploading segments... Aug 7 21:43:02.245849129 (12h3m23.98990599s) [upload=12439]: Upload finished. err: <nil> Duration: 4.063671385s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.XX.] duration: 934.938663ms FAIL (2025-08-07 21:45:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 934.977451ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [.................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 917.375602ms FAIL (2025-08-07 21:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 917.404019ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 40 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.XX] duration: 30.001355985s FAIL (2025-08-07 21:45:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001382091s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXX....] duration: 2.728359559s FAIL (2025-08-07 21:45:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== se=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces. Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments... Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:05.504575464 (12h1m27.248632324s) [upload=12408]: Starting upload Aug 7 21:41:05.504639224 (12h1m27.24869609s) [upload=12408]: Done starting upload Aug 7 21:41:05.504795449 (12h1m27.248852297s) [upload=12408]: Uploading segments... Aug 7 21:41:05.505027625 (12h1m27.249084472s) [upload=12408]: Got next segment. Inline: false Aug 7 21:41:05.505067645 (12h1m27.249124506s) [upload=12408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:05.505082107 (12h1m27.249138967s) [upload=12408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:35.236785438 (12h1m56.980842303s) [upload=12408]: Next segment err: context deadline exceeded Aug 7 21:41:35.236970291 (12h1m56.981027145s) [upload=12408]: Done uploading segments... Aug 7 21:41:35.236995163 (12h1m56.981052013s) [upload=12408]: Upload finished. err: context deadline exceeded Aug 7 21:41:35.644792324 (12h1m57.388849187s) [upload=12418]: Starting upload Aug 7 21:41:35.644872275 (12h1m57.388929138s) [upload=12418]: Done starting upload Aug 7 21:41:35.644961097 (12h1m57.389017947s) [upload=12418]: Uploading segments... Aug 7 21:41:35.645252533 (12h1m57.389309384s) [upload=12418]: Got next segment. Inline: false Aug 7 21:41:35.645298357 (12h1m57.389355205s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:35.645304084 (12h1m57.389360935s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:05.370939509 (12h2m27.114996372s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:42:05.370959077 (12h2m27.11501593s) [upload=12418]: Done uploading segments... Aug 7 21:42:05.370976831 (12h2m27.115033685s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:42:36.324068968 (12h2m58.068125815s) [upload=12431]: Starting upload Aug 7 21:42:36.324139117 (12h2m58.068195981s) [upload=12431]: Done starting upload Aug 7 21:42:36.324299395 (12h2m58.068356271s) [upload=12431]: Uploading segments... Aug 7 21:42:36.324485244 (12h2m58.068542225s) [upload=12431]: Got next segment. Inline: false Aug 7 21:42:36.324540099 (12h2m58.06859696s) [upload=12431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:36.324549943 (12h2m58.068606793s) [upload=12431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:43:06.057673777 (12h3m27.801730642s) [upload=12431]: Next segment err: context deadline exceeded Aug 7 21:43:06.057740059 (12h3m27.801796924s) [upload=12431]: Done uploading segments... Aug 7 21:43:06.057772188 (12h3m27.801829041s) [upload=12431]: Upload finished. err: context deadline exceeded Duration: 2.728411791s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [................................X............X.X.X..........] duration: 169.043688ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:45:34 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 169.06916ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 25 out of 60 is failed (threshold=3) [............................X.XXXXXXXXXXXXXXXXXXXXXXXX......] duration: 876.936144ms FAIL (2025-08-07 21:45:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Duration: 876.963426ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 2.841654153s FAIL (2025-08-07 21:45:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.841686394s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.......................XX.X..........X...] duration: 3.318549002s FAIL (2025-08-07 21:45:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 3 (12h3m20.890980883s) [upload=12439]: Starting upload Aug 7 21:42:59.146979817 (12h3m20.89103668s) [upload=12439]: Done starting upload Aug 7 21:42:59.147227284 (12h3m20.891284134s) [upload=12439]: Uploading segments... Aug 7 21:42:59.147266319 (12h3m20.891323188s) [upload=12439]: Got next segment. Inline: false Aug 7 21:42:59.147299756 (12h3m20.891356609s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:59.147305046 (12h3m20.891361896s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:59.534612872 (12h3m21.278669723s) [seg_pos={0 0}, upload=12439]: Begin upload segment... Aug 7 21:42:59.534692220 (12h3m21.278749069s) [seg_pos={0 0}, upload=12439]: Done begin upload segment. Aug 7 21:42:59.534758302 (12h3m21.278815173s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535350842 (12h3m21.279407711s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535933152 (12h3m21.279990018s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536298922 (12h3m21.280355769s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536689502 (12h3m21.280746369s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537035690 (12h3m21.281092562s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537373880 (12h3m21.281430743s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537716758 (12h3m21.281773618s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537995761 (12h3m21.282052656s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538051062 (12h3m21.282107912s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538386676 (12h3m21.282443539s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538674756 (12h3m21.282731616s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538946838 (12h3m21.283003698s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539233583 (12h3m21.283290444s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539738096 (12h3m21.28379497s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540187297 (12h3m21.284244165s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540518593 (12h3m21.284575456s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540774182 (12h3m21.284831043s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:43:00.120361086 (12h3m21.864417951s) [upload=12439]: Next returned nil segment Aug 7 21:43:00.120407815 (12h3m21.864464686s) [upload=12439]: Waiting for error group managing segments... Aug 7 21:43:00.271581920 (12h3m22.015638774s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.301379636 (12h3m22.0454365s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.315130147 (12h3m22.05918701s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.351162615 (12h3m22.095219465s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.383164149 (12h3m22.127221015s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.418050578 (12h3m22.162107442s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.451823428 (12h3m22.195880292s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.467192643 (12h3m22.211249504s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.495563998 (12h3m22.239620847s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.580244990 (12h3m22.324301836s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.647231825 (12h3m22.391288672s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.702946639 (12h3m22.447003485s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.748375718 (12h3m22.492432579s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801483011 (12h3m22.545539858s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801509099 (12h3m22.545565964s) [seg_pos={0 0}, upload=12439]: Segment reached optimal threshold of 14 pieces. Aug 7 21:43:00.801678905 (12h3m22.545735756s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802028506 (12h3m22.546085367s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802168197 (12h3m22.546225051s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802240477 (12h3m22.546297327s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: read tcp 139.59.144.22:42920->104.36.82.37:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:43:00.802254494 (12h3m22.546311343s) [seg_pos={0 0}, upload=12439]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:43:00.802379259 (12h3m22.546436112s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:43:00.802386197 (12h3m22.546443069s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.245820404 (12h3m23.98987728s) [upload=12439]: Done uploading segments... Aug 7 21:43:02.245849129 (12h3m23.98990599s) [upload=12439]: Upload finished. err: <nil> Duration: 3.318581469s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X....] duration: 1.1727174s FAIL (2025-08-07 21:45:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.172777725s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.................................X............X.X.X.........] duration: 106.569757ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:45:12 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 106.602795ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 2.75239586s FAIL (2025-08-07 21:45:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.752427017s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X.......................XX.X..........X..] duration: 3.295954889s FAIL (2025-08-07 21:45:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 3 (12h3m20.890980883s) [upload=12439]: Starting upload Aug 7 21:42:59.146979817 (12h3m20.89103668s) [upload=12439]: Done starting upload Aug 7 21:42:59.147227284 (12h3m20.891284134s) [upload=12439]: Uploading segments... Aug 7 21:42:59.147266319 (12h3m20.891323188s) [upload=12439]: Got next segment. Inline: false Aug 7 21:42:59.147299756 (12h3m20.891356609s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:59.147305046 (12h3m20.891361896s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:59.534612872 (12h3m21.278669723s) [seg_pos={0 0}, upload=12439]: Begin upload segment... Aug 7 21:42:59.534692220 (12h3m21.278749069s) [seg_pos={0 0}, upload=12439]: Done begin upload segment. Aug 7 21:42:59.534758302 (12h3m21.278815173s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535350842 (12h3m21.279407711s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535933152 (12h3m21.279990018s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536298922 (12h3m21.280355769s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536689502 (12h3m21.280746369s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537035690 (12h3m21.281092562s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537373880 (12h3m21.281430743s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537716758 (12h3m21.281773618s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537995761 (12h3m21.282052656s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538051062 (12h3m21.282107912s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538386676 (12h3m21.282443539s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538674756 (12h3m21.282731616s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538946838 (12h3m21.283003698s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539233583 (12h3m21.283290444s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539738096 (12h3m21.28379497s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540187297 (12h3m21.284244165s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540518593 (12h3m21.284575456s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540774182 (12h3m21.284831043s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:43:00.120361086 (12h3m21.864417951s) [upload=12439]: Next returned nil segment Aug 7 21:43:00.120407815 (12h3m21.864464686s) [upload=12439]: Waiting for error group managing segments... Aug 7 21:43:00.271581920 (12h3m22.015638774s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.301379636 (12h3m22.0454365s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.315130147 (12h3m22.05918701s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.351162615 (12h3m22.095219465s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.383164149 (12h3m22.127221015s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.418050578 (12h3m22.162107442s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.451823428 (12h3m22.195880292s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.467192643 (12h3m22.211249504s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.495563998 (12h3m22.239620847s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.580244990 (12h3m22.324301836s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.647231825 (12h3m22.391288672s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.702946639 (12h3m22.447003485s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.748375718 (12h3m22.492432579s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801483011 (12h3m22.545539858s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801509099 (12h3m22.545565964s) [seg_pos={0 0}, upload=12439]: Segment reached optimal threshold of 14 pieces. Aug 7 21:43:00.801678905 (12h3m22.545735756s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802028506 (12h3m22.546085367s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802168197 (12h3m22.546225051s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802240477 (12h3m22.546297327s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: read tcp 139.59.144.22:42920->104.36.82.37:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:43:00.802254494 (12h3m22.546311343s) [seg_pos={0 0}, upload=12439]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:43:00.802379259 (12h3m22.546436112s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:43:00.802386197 (12h3m22.546443069s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.245820404 (12h3m23.98987728s) [upload=12439]: Done uploading segments... Aug 7 21:43:02.245849129 (12h3m23.98990599s) [upload=12439]: Upload finished. err: <nil> Duration: 3.295991329s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [..............................XX..............XX.X..........] duration: 311.194952ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:45:08 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 311.216743ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 858.812729ms FAIL (2025-08-07 21:45:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 858.850019ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 11.979231553s FAIL (2025-08-07 21:45:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.979248206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [..................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 3.35766897s FAIL (2025-08-07 21:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.357701604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 39 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.X] duration: 30.001280095s FAIL (2025-08-07 21:45:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001311581s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXX...] duration: 3.710334194s FAIL (2025-08-07 21:44:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== se=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces. Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments... Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:05.504575464 (12h1m27.248632324s) [upload=12408]: Starting upload Aug 7 21:41:05.504639224 (12h1m27.24869609s) [upload=12408]: Done starting upload Aug 7 21:41:05.504795449 (12h1m27.248852297s) [upload=12408]: Uploading segments... Aug 7 21:41:05.505027625 (12h1m27.249084472s) [upload=12408]: Got next segment. Inline: false Aug 7 21:41:05.505067645 (12h1m27.249124506s) [upload=12408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:05.505082107 (12h1m27.249138967s) [upload=12408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:35.236785438 (12h1m56.980842303s) [upload=12408]: Next segment err: context deadline exceeded Aug 7 21:41:35.236970291 (12h1m56.981027145s) [upload=12408]: Done uploading segments... Aug 7 21:41:35.236995163 (12h1m56.981052013s) [upload=12408]: Upload finished. err: context deadline exceeded Aug 7 21:41:35.644792324 (12h1m57.388849187s) [upload=12418]: Starting upload Aug 7 21:41:35.644872275 (12h1m57.388929138s) [upload=12418]: Done starting upload Aug 7 21:41:35.644961097 (12h1m57.389017947s) [upload=12418]: Uploading segments... Aug 7 21:41:35.645252533 (12h1m57.389309384s) [upload=12418]: Got next segment. Inline: false Aug 7 21:41:35.645298357 (12h1m57.389355205s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:35.645304084 (12h1m57.389360935s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:05.370939509 (12h2m27.114996372s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:42:05.370959077 (12h2m27.11501593s) [upload=12418]: Done uploading segments... Aug 7 21:42:05.370976831 (12h2m27.115033685s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:42:36.324068968 (12h2m58.068125815s) [upload=12431]: Starting upload Aug 7 21:42:36.324139117 (12h2m58.068195981s) [upload=12431]: Done starting upload Aug 7 21:42:36.324299395 (12h2m58.068356271s) [upload=12431]: Uploading segments... Aug 7 21:42:36.324485244 (12h2m58.068542225s) [upload=12431]: Got next segment. Inline: false Aug 7 21:42:36.324540099 (12h2m58.06859696s) [upload=12431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:36.324549943 (12h2m58.068606793s) [upload=12431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:43:06.057673777 (12h3m27.801730642s) [upload=12431]: Next segment err: context deadline exceeded Aug 7 21:43:06.057740059 (12h3m27.801796924s) [upload=12431]: Done uploading segments... Aug 7 21:43:06.057772188 (12h3m27.801829041s) [upload=12431]: Upload finished. err: context deadline exceeded Duration: 3.710377962s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 1m0.001400692s FAIL (2025-08-07 21:44:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001423779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 25 out of 60 is failed (threshold=3) [.............................X.XXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 12.204249022s FAIL (2025-08-07 21:44:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Duration: 12.2042804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [...................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 10.375306584s FAIL (2025-08-07 21:44:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.375334002s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [..................................X............X.X.X........] duration: 242.068655ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:44:30 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 242.097774ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [..............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 2.61778079s FAIL (2025-08-07 21:44:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.617807705s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [...............................XX..............XX.X.........] duration: 328.548132ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:44:28 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 328.606213ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...] duration: 1.011928477s FAIL (2025-08-07 21:44:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.011951979s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 908.226573ms FAIL (2025-08-07 21:44:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 908.256065ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X.......................XX.X..........X.] duration: 2.853773378s FAIL (2025-08-07 21:44:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 3 (12h3m20.890980883s) [upload=12439]: Starting upload Aug 7 21:42:59.146979817 (12h3m20.89103668s) [upload=12439]: Done starting upload Aug 7 21:42:59.147227284 (12h3m20.891284134s) [upload=12439]: Uploading segments... Aug 7 21:42:59.147266319 (12h3m20.891323188s) [upload=12439]: Got next segment. Inline: false Aug 7 21:42:59.147299756 (12h3m20.891356609s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:59.147305046 (12h3m20.891361896s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:59.534612872 (12h3m21.278669723s) [seg_pos={0 0}, upload=12439]: Begin upload segment... Aug 7 21:42:59.534692220 (12h3m21.278749069s) [seg_pos={0 0}, upload=12439]: Done begin upload segment. Aug 7 21:42:59.534758302 (12h3m21.278815173s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535350842 (12h3m21.279407711s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535933152 (12h3m21.279990018s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536298922 (12h3m21.280355769s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536689502 (12h3m21.280746369s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537035690 (12h3m21.281092562s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537373880 (12h3m21.281430743s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537716758 (12h3m21.281773618s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537995761 (12h3m21.282052656s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538051062 (12h3m21.282107912s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538386676 (12h3m21.282443539s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538674756 (12h3m21.282731616s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538946838 (12h3m21.283003698s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539233583 (12h3m21.283290444s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539738096 (12h3m21.28379497s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540187297 (12h3m21.284244165s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540518593 (12h3m21.284575456s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540774182 (12h3m21.284831043s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:43:00.120361086 (12h3m21.864417951s) [upload=12439]: Next returned nil segment Aug 7 21:43:00.120407815 (12h3m21.864464686s) [upload=12439]: Waiting for error group managing segments... Aug 7 21:43:00.271581920 (12h3m22.015638774s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.301379636 (12h3m22.0454365s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.315130147 (12h3m22.05918701s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.351162615 (12h3m22.095219465s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.383164149 (12h3m22.127221015s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.418050578 (12h3m22.162107442s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.451823428 (12h3m22.195880292s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.467192643 (12h3m22.211249504s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.495563998 (12h3m22.239620847s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.580244990 (12h3m22.324301836s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.647231825 (12h3m22.391288672s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.702946639 (12h3m22.447003485s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.748375718 (12h3m22.492432579s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801483011 (12h3m22.545539858s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801509099 (12h3m22.545565964s) [seg_pos={0 0}, upload=12439]: Segment reached optimal threshold of 14 pieces. Aug 7 21:43:00.801678905 (12h3m22.545735756s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802028506 (12h3m22.546085367s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802168197 (12h3m22.546225051s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802240477 (12h3m22.546297327s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: read tcp 139.59.144.22:42920->104.36.82.37:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:43:00.802254494 (12h3m22.546311343s) [seg_pos={0 0}, upload=12439]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:43:00.802379259 (12h3m22.546436112s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:43:00.802386197 (12h3m22.546443069s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.245820404 (12h3m23.98987728s) [upload=12439]: Done uploading segments... Aug 7 21:43:02.245849129 (12h3m23.98990599s) [upload=12439]: Upload finished. err: <nil> Duration: 2.853802621s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [...................................X............X.X.X.......] duration: 113.241644ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:44:07 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 113.27144ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [...............X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 2.858746044s FAIL (2025-08-07 21:44:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.858773685s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXX..] duration: 11.900328485s FAIL (2025-08-07 21:44:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== se=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces. Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments... Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:05.504575464 (12h1m27.248632324s) [upload=12408]: Starting upload Aug 7 21:41:05.504639224 (12h1m27.24869609s) [upload=12408]: Done starting upload Aug 7 21:41:05.504795449 (12h1m27.248852297s) [upload=12408]: Uploading segments... Aug 7 21:41:05.505027625 (12h1m27.249084472s) [upload=12408]: Got next segment. Inline: false Aug 7 21:41:05.505067645 (12h1m27.249124506s) [upload=12408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:05.505082107 (12h1m27.249138967s) [upload=12408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:35.236785438 (12h1m56.980842303s) [upload=12408]: Next segment err: context deadline exceeded Aug 7 21:41:35.236970291 (12h1m56.981027145s) [upload=12408]: Done uploading segments... Aug 7 21:41:35.236995163 (12h1m56.981052013s) [upload=12408]: Upload finished. err: context deadline exceeded Aug 7 21:41:35.644792324 (12h1m57.388849187s) [upload=12418]: Starting upload Aug 7 21:41:35.644872275 (12h1m57.388929138s) [upload=12418]: Done starting upload Aug 7 21:41:35.644961097 (12h1m57.389017947s) [upload=12418]: Uploading segments... Aug 7 21:41:35.645252533 (12h1m57.389309384s) [upload=12418]: Got next segment. Inline: false Aug 7 21:41:35.645298357 (12h1m57.389355205s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:35.645304084 (12h1m57.389360935s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:05.370939509 (12h2m27.114996372s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:42:05.370959077 (12h2m27.11501593s) [upload=12418]: Done uploading segments... Aug 7 21:42:05.370976831 (12h2m27.115033685s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:42:36.324068968 (12h2m58.068125815s) [upload=12431]: Starting upload Aug 7 21:42:36.324139117 (12h2m58.068195981s) [upload=12431]: Done starting upload Aug 7 21:42:36.324299395 (12h2m58.068356271s) [upload=12431]: Uploading segments... Aug 7 21:42:36.324485244 (12h2m58.068542225s) [upload=12431]: Got next segment. Inline: false Aug 7 21:42:36.324540099 (12h2m58.06859696s) [upload=12431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:36.324549943 (12h2m58.068606793s) [upload=12431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:43:06.057673777 (12h3m27.801730642s) [upload=12431]: Next segment err: context deadline exceeded Aug 7 21:43:06.057740059 (12h3m27.801796924s) [upload=12431]: Done uploading segments... Aug 7 21:43:06.057772188 (12h3m27.801829041s) [upload=12431]: Upload finished. err: context deadline exceeded Duration: 11.900365625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X.] duration: 921.219371ms FAIL (2025-08-07 21:43:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 921.250876ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 4.799689731s FAIL (2025-08-07 21:43:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.799715852s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 38 out of 60 is failed (threshold=3) [....................XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 23.573205354s FAIL (2025-08-07 21:43:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng upload Aug 7 21:34:39.735850106 (11h55m1.479906957s) [upload=12318]: Uploading segments... Aug 7 21:34:39.735909766 (11h55m1.479966635s) [upload=12318]: Got next segment. Inline: true Aug 7 21:34:39.735915314 (11h55m1.479972174s) [upload=12318]: Waiting for error group managing segments... Aug 7 21:34:39.736018840 (11h55m1.480075701s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:39.736024699 (11h55m1.480081559s) [upload=12318]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:39.736028031 (11h55m1.480084892s) [upload=12318]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:09.464049958 (11h55m31.208106818s) [upload=12318]: Done uploading segments... Aug 7 21:35:09.464072716 (11h55m31.208129576s) [upload=12318]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:34.656807174 (11h55m56.400864101s) [upload=12328]: Starting upload Aug 7 21:35:34.656894285 (11h55m56.400951152s) [upload=12328]: Done starting upload Aug 7 21:35:34.656967676 (11h55m56.401024525s) [upload=12328]: Uploading segments... Aug 7 21:35:34.657016134 (11h55m56.401073003s) [upload=12328]: Got next segment. Inline: true Aug 7 21:35:34.657021131 (11h55m56.401077993s) [upload=12328]: Waiting for error group managing segments... Aug 7 21:35:34.657065881 (11h55m56.401122731s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:34.657071386 (11h55m56.401128236s) [upload=12328]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:34.657074624 (11h55m56.401131484s) [upload=12328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:04.380124890 (11h56m26.124181742s) [upload=12328]: Done uploading segments... Aug 7 21:36:04.380153582 (11h56m26.124210448s) [upload=12328]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:04.798191174 (11h56m26.542248041s) [upload=12340]: Starting upload Aug 7 21:36:04.798286128 (11h56m26.542342995s) [upload=12340]: Done starting upload Aug 7 21:36:04.798372008 (11h56m26.542428871s) [upload=12340]: Uploading segments... Aug 7 21:36:04.798434433 (11h56m26.542491304s) [upload=12340]: Got next segment. Inline: true Aug 7 21:36:04.798441202 (11h56m26.542498066s) [upload=12340]: Waiting for error group managing segments... Aug 7 21:36:04.798496298 (11h56m26.542553165s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:04.798502276 (11h56m26.542559141s) [upload=12340]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:04.798505608 (11h56m26.542562472s) [upload=12340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:34.510180222 (11h56m56.254237082s) [upload=12340]: Done uploading segments... Aug 7 21:36:34.510201902 (11h56m56.254258762s) [upload=12340]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:16.815303797 (11h57m38.559360649s) [upload=12354]: Starting upload Aug 7 21:37:16.815396631 (11h57m38.559453502s) [upload=12354]: Done starting upload Aug 7 21:37:16.815743613 (11h57m38.559800465s) [upload=12354]: Uploading segments... Aug 7 21:37:16.815841404 (11h57m38.559898275s) [upload=12354]: Got next segment. Inline: true Aug 7 21:37:16.815849471 (11h57m38.559906335s) [upload=12354]: Waiting for error group managing segments... Aug 7 21:37:16.815913502 (11h57m38.559970351s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:16.815919848 (11h57m38.559976709s) [upload=12354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:16.815924222 (11h57m38.559981087s) [upload=12354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:46.539729655 (11h58m8.283786508s) [upload=12354]: Done uploading segments... Aug 7 21:37:46.539785208 (11h58m8.283842069s) [upload=12354]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:27.592061643 (11h58m49.336118515s) [upload=12370]: Starting upload Aug 7 21:38:27.592133721 (11h58m49.336190591s) [upload=12370]: Done starting upload Aug 7 21:38:27.592194600 (11h58m49.336251462s) [upload=12370]: Uploading segments... Aug 7 21:38:27.592243475 (11h58m49.336300347s) [upload=12370]: Got next segment. Inline: true Aug 7 21:38:27.592249638 (11h58m49.336306506s) [upload=12370]: Waiting for error group managing segments... Aug 7 21:38:27.592301586 (11h58m49.336358453s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:27.592307915 (11h58m49.33636478s) [upload=12370]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:27.592314515 (11h58m49.336371384s) [upload=12370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:57.295172349 (11h59m19.039229199s) [upload=12370]: Done uploading segments... Aug 7 21:38:57.295199050 (11h59m19.0392559s) [upload=12370]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:57.705852359 (11h59m19.449909219s) [upload=12378]: Starting upload Aug 7 21:38:57.705946659 (11h59m19.450003522s) [upload=12378]: Done starting upload Aug 7 21:38:57.706046021 (11h59m19.450102881s) [upload=12378]: Uploading segments... Aug 7 21:38:57.706107301 (11h59m19.450164171s) [upload=12378]: Got next segment. Inline: true Aug 7 21:38:57.706112988 (11h59m19.450169848s) [upload=12378]: Waiting for error group managing segments... Aug 7 21:38:57.706170165 (11h59m19.450227028s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:57.706175372 (11h59m19.450232233s) [upload=12378]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:57.706178976 (11h59m19.450235838s) [upload=12378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:27.434609736 (11h59m49.178666597s) [upload=12378]: Done uploading segments... Aug 7 21:39:27.434644375 (11h59m49.178701235s) [upload=12378]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:56.317558286 (12h0m18.061615147s) [upload=12392]: Starting upload Aug 7 21:39:56.317614127 (12h0m18.061670988s) [upload=12392]: Done starting upload Aug 7 21:39:56.317683655 (12h0m18.061740509s) [upload=12392]: Uploading segments... Aug 7 21:39:56.317748361 (12h0m18.061805241s) [upload=12392]: Got next segment. Inline: true Aug 7 21:39:56.317754805 (12h0m18.061811669s) [upload=12392]: Waiting for error group managing segments... Aug 7 21:39:56.317831053 (12h0m18.061887914s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:56.317837732 (12h0m18.061894593s) [upload=12392]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:56.317841055 (12h0m18.061897915s) [upload=12392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:26.032549829 (12h0m47.776606689s) [upload=12392]: Done uploading segments... Aug 7 21:40:26.032570897 (12h0m47.776627757s) [upload=12392]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:41.354808211 (12h1m3.098865072s) [upload=12402]: Starting upload Aug 7 21:40:41.354938701 (12h1m3.098995568s) [upload=12402]: Done starting upload Aug 7 21:40:41.355036214 (12h1m3.099093064s) [upload=12402]: Uploading segments... Aug 7 21:40:41.355109185 (12h1m3.099166055s) [upload=12402]: Got next segment. Inline: true Aug 7 21:40:41.355116835 (12h1m3.0991737s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:40:41.355183407 (12h1m3.09924027s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:41.355190433 (12h1m3.099247297s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:41.355194357 (12h1m3.09925122s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:11.078938147 (12h1m32.822994999s) [upload=12402]: Done uploading segments... Aug 7 21:41:11.078958711 (12h1m32.823015562s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.899938086 (12h1m56.643994947s) [upload=12417]: Starting upload Aug 7 21:41:34.900014273 (12h1m56.644071133s) [upload=12417]: Done starting upload Aug 7 21:41:34.900081071 (12h1m56.644137926s) [upload=12417]: Uploading segments... Aug 7 21:41:34.900153322 (12h1m56.644210172s) [upload=12417]: Got next segment. Inline: true Aug 7 21:41:34.900158578 (12h1m56.644215438s) [upload=12417]: Waiting for error group managing segments... Aug 7 21:41:34.900228776 (12h1m56.644285623s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.900237327 (12h1m56.644294194s) [upload=12417]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.900241831 (12h1m56.644298696s) [upload=12417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:04.627262793 (12h2m26.371319653s) [upload=12417]: Done uploading segments... Aug 7 21:42:04.627301743 (12h2m26.371358604s) [upload=12417]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:18.477725352 (12h2m40.221782248s) [upload=12428]: Starting upload Aug 7 21:42:18.477815072 (12h2m40.221871935s) [upload=12428]: Done starting upload Aug 7 21:42:18.477903321 (12h2m40.221960189s) [upload=12428]: Uploading segments... Aug 7 21:42:18.478047763 (12h2m40.222104633s) [upload=12428]: Got next segment. Inline: true Aug 7 21:42:18.478056185 (12h2m40.222113051s) [upload=12428]: Waiting for error group managing segments... Aug 7 21:42:18.478131478 (12h2m40.222188328s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:18.478138171 (12h2m40.222195021s) [upload=12428]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:18.478144895 (12h2m40.222201761s) [upload=12428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:48.201994950 (12h3m9.946051801s) [upload=12428]: Done uploading segments... Aug 7 21:42:48.202028187 (12h3m9.946085036s) [upload=12428]: Upload finished. err: metaclient: context deadline exceeded Duration: 23.573240172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 25 out of 60 is failed (threshold=3) [..............................X.XXXXXXXXXXXXXXXXXXXXXXXX....] duration: 924.403882ms FAIL (2025-08-07 21:43:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Duration: 924.442678ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 5 out of 60 is failed (threshold=3) [................................XX..............XX.X........] duration: 321.589545ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:43:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 321.627896ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXX.] duration: 2.504246889s FAIL (2025-08-07 21:43:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== se=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces. Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments... Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:05.504575464 (12h1m27.248632324s) [upload=12408]: Starting upload Aug 7 21:41:05.504639224 (12h1m27.24869609s) [upload=12408]: Done starting upload Aug 7 21:41:05.504795449 (12h1m27.248852297s) [upload=12408]: Uploading segments... Aug 7 21:41:05.505027625 (12h1m27.249084472s) [upload=12408]: Got next segment. Inline: false Aug 7 21:41:05.505067645 (12h1m27.249124506s) [upload=12408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:05.505082107 (12h1m27.249138967s) [upload=12408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:35.236785438 (12h1m56.980842303s) [upload=12408]: Next segment err: context deadline exceeded Aug 7 21:41:35.236970291 (12h1m56.981027145s) [upload=12408]: Done uploading segments... Aug 7 21:41:35.236995163 (12h1m56.981052013s) [upload=12408]: Upload finished. err: context deadline exceeded Aug 7 21:41:35.644792324 (12h1m57.388849187s) [upload=12418]: Starting upload Aug 7 21:41:35.644872275 (12h1m57.388929138s) [upload=12418]: Done starting upload Aug 7 21:41:35.644961097 (12h1m57.389017947s) [upload=12418]: Uploading segments... Aug 7 21:41:35.645252533 (12h1m57.389309384s) [upload=12418]: Got next segment. Inline: false Aug 7 21:41:35.645298357 (12h1m57.389355205s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:35.645304084 (12h1m57.389360935s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:05.370939509 (12h2m27.114996372s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:42:05.370959077 (12h2m27.11501593s) [upload=12418]: Done uploading segments... Aug 7 21:42:05.370976831 (12h2m27.115033685s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:42:36.324068968 (12h2m58.068125815s) [upload=12431]: Starting upload Aug 7 21:42:36.324139117 (12h2m58.068195981s) [upload=12431]: Done starting upload Aug 7 21:42:36.324299395 (12h2m58.068356271s) [upload=12431]: Uploading segments... Aug 7 21:42:36.324485244 (12h2m58.068542225s) [upload=12431]: Got next segment. Inline: false Aug 7 21:42:36.324540099 (12h2m58.06859696s) [upload=12431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:36.324549943 (12h2m58.068606793s) [upload=12431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:43:06.057673777 (12h3m27.801730642s) [upload=12431]: Next segment err: context deadline exceeded Aug 7 21:43:06.057740059 (12h3m27.801796924s) [upload=12431]: Done uploading segments... Aug 7 21:43:06.057772188 (12h3m27.801829041s) [upload=12431]: Upload finished. err: context deadline exceeded Duration: 2.504277573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 38 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X] duration: 30.000467382s FAIL (2025-08-07 21:43:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000483848s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [....................................X............X.X.X......] duration: 105.916314ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:43:23 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 105.956215ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 3.326686149s FAIL (2025-08-07 21:43:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.326711889s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..] duration: 2.354164544s FAIL (2025-08-07 21:43:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.354252622s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 949.874824ms FAIL (2025-08-07 21:43:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 949.908959ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.002354577s FAIL (2025-08-07 21:43:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== se=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921441470 (11h59m56.665498332s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.921487466 (11h59m56.665544328s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.922317331 (11h59m56.666374206s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923026433 (11h59m56.667083296s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923407595 (11h59m56.667464455s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.923763535 (11h59m56.667820401s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924101773 (11h59m56.668158645s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924425635 (11h59m56.668482498s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.924925232 (11h59m56.668982102s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:34.925233497 (11h59m56.669290362s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Uploading piece... Aug 7 21:39:35.531363219 (11h59m57.275420069s) [upload=12385]: Next returned nil segment Aug 7 21:39:35.531379805 (11h59m57.275436664s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:39:35.709218321 (11h59m57.453275183s) [noise=true , address=104.36.82.16:4294 , piece_id=OFVD5V2JJNQF6HGZDN7P2JXRNAAT6EWL67VD4HRPGM36SWKHI2CQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.747949042 (11h59m57.492005889s) [noise=true , address=45.91.116.8:4048 , piece_id=WWF6JJIOTPJIKOBZXVC7FRVTDSIPLRIKNN3BDJABOPOBAT7GMHZQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.748418025 (11h59m57.492474871s) [noise=true , address=104.36.82.14:4222 , piece_id=3QK6KKHJKKEFLV424LSEVYXOP63K5MTQ3YDREYFKWXFZ7ZUEO2WA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.751190468 (11h59m57.495247318s) [noise=true , address=45.91.116.8:4045 , piece_id=4QJK7SQYXCNSGZF6SL43PF5LKH74TQT2PU5JGIXXAYRQHDGRUNRA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754282840 (11h59m57.498339696s) [noise=true , address=104.36.82.8:4969 , piece_id=Z3CTCWXCOEUNE6JKJMKDKLRN2XMBQTLGDMSH4NYXYFREOQC4NEJA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.754668188 (11h59m57.498725035s) [noise=true , address=45.91.116.9:4261 , piece_id=D2J6FNBWOIHWV5UWJF33T6DCVW4WBHTOH5WH3KEAH7UOD45S6IMA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.760515970 (11h59m57.50457282s) [noise=true , address=45.91.116.13:4153 , piece_id=VPLNW4ELDDHLPEDQ2HVWPSIBC45XN7ORVLFG75FDALY7AFYRYNEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.765275771 (11h59m57.509332626s) [noise=true , address=77.90.31.18:4806 , piece_id=5VFMZTJ4UNYFVMRNXJUAAQXTOWT7GOJWM5LKVQYB4J2I6FWXY26Q, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.775261406 (11h59m57.519318257s) [noise=true , address=45.91.116.5:4298 , piece_id=AY2EZFYLXMYKDJXBHFZ2TGP3MDVKMMQMF3TO4ZYGEP4DNYBNI3RA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.794687407 (11h59m57.538744255s) [noise=true , address=45.91.116.14:4189 , piece_id=4RJKE6N53OI4SFMVQG5AZXEBRBBJKZLGU4P7JGLXI4QFYOWGQB6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835050406 (11h59m57.579107253s) [noise=true , address=45.91.116.11:4119 , piece_id=ZB724UKE5HE7KPX7NPI53UVETXJLCOIYLTIPEEPOQERIHRSNMV5A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.835339286 (11h59m57.579396147s) [noise=true , address=77.90.31.16:4735 , piece_id=2FGRTS7QNZVAUNMIJ4OUJGE4R2KWFY3DKLJYDRXVL5AHNR7XXTFQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853563302 (11h59m57.597620154s) [noise=true , address=45.91.116.12:4227 , piece_id=QVRKE7V6TCIRXR7W2UFXQ2CCNLBTPNJMUEYIDOHFKFZQN4MGLY6A, seg_pos={0 0}, upload=12385]: Done uploading piece. err: <nil> Aug 7 21:39:35.853590381 (11h59m57.597647242s) [seg_pos={0 0}, upload=12385]: Segment reached optimal threshold of 13 pieces. Aug 7 21:39:35.853877651 (11h59m57.597934499s) [noise=true , address=77.90.31.21:4878 , piece_id=PK3NUCJC32GLOGCFETGQPWFMQ3KDI3WPAZTIRQ6IMDIPD7WSHSEQ, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854014198 (11h59m57.598071065s) [noise=true , address=45.91.116.6:4974 , piece_id=J3MMVSDVGWV7CH56IPZYPOLPJ6FYZADXICDBZPEL532HRBCYAGYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 21:39:35.854095786 (11h59m57.598152646s) [noise=true , address=104.36.82.2:4732 , piece_id=PLZN5HJHBGPCX2733R6CPV4TDAP47FFR5U7ORXKCVGM46DTQZSYA, seg_pos={0 0}, upload=12385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38226->104.36.82.2:4732: use of closed network connection Aug 7 21:39:35.854123369 (11h59m57.59818023s) [seg_pos={0 0}, upload=12385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:39:35.854214014 (11h59m57.598270883s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:39:35.854220398 (11h59m57.598277266s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:05.070835032 (12h1m26.814891887s) [upload=12385]: Done uploading segments... Aug 7 21:41:05.070862625 (12h1m26.814919475s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:05.504575464 (12h1m27.248632324s) [upload=12408]: Starting upload Aug 7 21:41:05.504639224 (12h1m27.24869609s) [upload=12408]: Done starting upload Aug 7 21:41:05.504795449 (12h1m27.248852297s) [upload=12408]: Uploading segments... Aug 7 21:41:05.505027625 (12h1m27.249084472s) [upload=12408]: Got next segment. Inline: false Aug 7 21:41:05.505067645 (12h1m27.249124506s) [upload=12408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:05.505082107 (12h1m27.249138967s) [upload=12408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:35.236785438 (12h1m56.980842303s) [upload=12408]: Next segment err: context deadline exceeded Aug 7 21:41:35.236970291 (12h1m56.981027145s) [upload=12408]: Done uploading segments... Aug 7 21:41:35.236995163 (12h1m56.981052013s) [upload=12408]: Upload finished. err: context deadline exceeded Aug 7 21:41:35.644792324 (12h1m57.388849187s) [upload=12418]: Starting upload Aug 7 21:41:35.644872275 (12h1m57.388929138s) [upload=12418]: Done starting upload Aug 7 21:41:35.644961097 (12h1m57.389017947s) [upload=12418]: Uploading segments... Aug 7 21:41:35.645252533 (12h1m57.389309384s) [upload=12418]: Got next segment. Inline: false Aug 7 21:41:35.645298357 (12h1m57.389355205s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:35.645304084 (12h1m57.389360935s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:05.370939509 (12h2m27.114996372s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:42:05.370959077 (12h2m27.11501593s) [upload=12418]: Done uploading segments... Aug 7 21:42:05.370976831 (12h2m27.115033685s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:42:36.324068968 (12h2m58.068125815s) [upload=12431]: Starting upload Aug 7 21:42:36.324139117 (12h2m58.068195981s) [upload=12431]: Done starting upload Aug 7 21:42:36.324299395 (12h2m58.068356271s) [upload=12431]: Uploading segments... Aug 7 21:42:36.324485244 (12h2m58.068542225s) [upload=12431]: Got next segment. Inline: false Aug 7 21:42:36.324540099 (12h2m58.06859696s) [upload=12431]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:36.324549943 (12h2m58.068606793s) [upload=12431]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:43:06.057673777 (12h3m27.801730642s) [upload=12431]: Next segment err: context deadline exceeded Aug 7 21:43:06.057740059 (12h3m27.801796924s) [upload=12431]: Done uploading segments... Aug 7 21:43:06.057772188 (12h3m27.801829041s) [upload=12431]: Upload finished. err: context deadline exceeded Duration: 30.002379745s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X............X.X.X.....] duration: 120.567114ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:43:06 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 120.614846ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 25 out of 60 is failed (threshold=3) [...............................X.XXXXXXXXXXXXXXXXXXXXXXXX...] duration: 11.917758029s FAIL (2025-08-07 21:43:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.162032485 (11h54m52.906089348s) [upload=12316]: Starting upload Aug 7 21:34:31.162129373 (11h54m52.906186241s) [upload=12316]: Done starting upload Aug 7 21:34:31.162197010 (11h54m52.906253877s) [upload=12316]: Uploading segments... Aug 7 21:34:31.162265332 (11h54m52.906322201s) [upload=12316]: Got next segment. Inline: true Aug 7 21:34:31.162271192 (11h54m52.906328063s) [upload=12316]: Waiting for error group managing segments... Aug 7 21:34:31.162343122 (11h54m52.906399986s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.162350473 (11h54m52.906407351s) [upload=12316]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.162357235 (11h54m52.906414101s) [upload=12316]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.893978311 (11h55m52.638035172s) [upload=12316]: Done uploading segments... Aug 7 21:35:30.894001661 (11h55m52.638058513s) [upload=12316]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:44.593281832 (11h56m6.337338681s) [upload=12332]: Starting upload Aug 7 21:35:44.593379702 (11h56m6.337436562s) [upload=12332]: Done starting upload Aug 7 21:35:44.593440159 (11h56m6.33749701s) [upload=12332]: Uploading segments... Aug 7 21:35:44.593541679 (11h56m6.337598542s) [upload=12332]: Got next segment. Inline: true Aug 7 21:35:44.593548645 (11h56m6.337605505s) [upload=12332]: Waiting for error group managing segments... Aug 7 21:35:44.593622380 (11h56m6.337679229s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:44.593628916 (11h56m6.337685776s) [upload=12332]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:44.593632768 (11h56m6.337689686s) [upload=12332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:44.320915363 (11h57m6.064972224s) [upload=12332]: Done uploading segments... Aug 7 21:36:44.320936819 (11h57m6.06499367s) [upload=12332]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:44.737096160 (11h57m6.48115302s) [upload=12350]: Starting upload Aug 7 21:36:44.737199233 (11h57m6.481256079s) [upload=12350]: Done starting upload Aug 7 21:36:44.737256356 (11h57m6.481313202s) [upload=12350]: Uploading segments... Aug 7 21:36:44.737309397 (11h57m6.481366267s) [upload=12350]: Got next segment. Inline: true Aug 7 21:36:44.737314124 (11h57m6.481370986s) [upload=12350]: Waiting for error group managing segments... Aug 7 21:36:44.737363580 (11h57m6.481420443s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.737369794 (11h57m6.481426664s) [upload=12350]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:44.737374614 (11h57m6.481431481s) [upload=12350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.468181532 (11h58m6.212238379s) [upload=12350]: Done uploading segments... Aug 7 21:37:44.468202662 (11h58m6.212259509s) [upload=12350]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:44.877004541 (11h58m6.621061411s) [upload=12363]: Starting upload Aug 7 21:37:44.877093129 (11h58m6.621150048s) [upload=12363]: Done starting upload Aug 7 21:37:44.877179828 (11h58m6.621236677s) [upload=12363]: Uploading segments... Aug 7 21:37:44.877253719 (11h58m6.621310589s) [upload=12363]: Got next segment. Inline: true Aug 7 21:37:44.877262492 (11h58m6.621319365s) [upload=12363]: Waiting for error group managing segments... Aug 7 21:37:44.877313668 (11h58m6.621370528s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:44.877319037 (11h58m6.621375897s) [upload=12363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:44.877322512 (11h58m6.621379375s) [upload=12363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:44.603781017 (11h59m6.347837867s) [upload=12363]: Done uploading segments... Aug 7 21:38:44.603802220 (11h59m6.347859069s) [upload=12363]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:45.032933533 (11h59m6.776990379s) [upload=12375]: Starting upload Aug 7 21:38:45.033010670 (11h59m6.777067533s) [upload=12375]: Done starting upload Aug 7 21:38:45.033084965 (11h59m6.777141811s) [upload=12375]: Uploading segments... Aug 7 21:38:45.033156998 (11h59m6.777213868s) [upload=12375]: Got next segment. Inline: true Aug 7 21:38:45.033162279 (11h59m6.777219127s) [upload=12375]: Waiting for error group managing segments... Aug 7 21:38:45.033211860 (11h59m6.777268707s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:45.033216660 (11h59m6.77727352s) [upload=12375]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:45.033219329 (11h59m6.777276175s) [upload=12375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:44.758881739 (12h0m6.502938587s) [upload=12375]: Done uploading segments... Aug 7 21:39:44.758902501 (12h0m6.502959346s) [upload=12375]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:45.185696259 (12h0m6.929753109s) [upload=12388]: Starting upload Aug 7 21:39:45.185761209 (12h0m6.929818072s) [upload=12388]: Done starting upload Aug 7 21:39:45.185834033 (12h0m6.929890896s) [upload=12388]: Uploading segments... Aug 7 21:39:45.185982126 (12h0m6.930038997s) [upload=12388]: Got next segment. Inline: true Aug 7 21:39:45.185990859 (12h0m6.930047726s) [upload=12388]: Waiting for error group managing segments... Aug 7 21:39:45.186067966 (12h0m6.930124827s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:45.186074972 (12h0m6.930131844s) [upload=12388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:45.186078706 (12h0m6.930135567s) [upload=12388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:44.901948337 (12h1m6.646005185s) [upload=12388]: Done uploading segments... Aug 7 21:40:44.901969954 (12h1m6.646026803s) [upload=12388]: Upload finished. err: metaclient: context deadline exceeded Duration: 11.917789562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X.......................XX.X..........X] duration: 4.166730325s FAIL (2025-08-07 21:43:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 3 (12h3m20.890980883s) [upload=12439]: Starting upload Aug 7 21:42:59.146979817 (12h3m20.89103668s) [upload=12439]: Done starting upload Aug 7 21:42:59.147227284 (12h3m20.891284134s) [upload=12439]: Uploading segments... Aug 7 21:42:59.147266319 (12h3m20.891323188s) [upload=12439]: Got next segment. Inline: false Aug 7 21:42:59.147299756 (12h3m20.891356609s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:59.147305046 (12h3m20.891361896s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:42:59.534612872 (12h3m21.278669723s) [seg_pos={0 0}, upload=12439]: Begin upload segment... Aug 7 21:42:59.534692220 (12h3m21.278749069s) [seg_pos={0 0}, upload=12439]: Done begin upload segment. Aug 7 21:42:59.534758302 (12h3m21.278815173s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535350842 (12h3m21.279407711s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.535933152 (12h3m21.279990018s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536298922 (12h3m21.280355769s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.536689502 (12h3m21.280746369s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537035690 (12h3m21.281092562s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537373880 (12h3m21.281430743s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537716758 (12h3m21.281773618s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.537995761 (12h3m21.282052656s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538051062 (12h3m21.282107912s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538386676 (12h3m21.282443539s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538674756 (12h3m21.282731616s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.538946838 (12h3m21.283003698s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539233583 (12h3m21.283290444s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.539738096 (12h3m21.28379497s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540187297 (12h3m21.284244165s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540518593 (12h3m21.284575456s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:42:59.540774182 (12h3m21.284831043s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Uploading piece... Aug 7 21:43:00.120361086 (12h3m21.864417951s) [upload=12439]: Next returned nil segment Aug 7 21:43:00.120407815 (12h3m21.864464686s) [upload=12439]: Waiting for error group managing segments... Aug 7 21:43:00.271581920 (12h3m22.015638774s) [noise=true , address=104.36.82.36:4632 , piece_id=WCZEM2JMOB3WL6RJJ5MVXPP6TWFEZ3TSP7DOX33JHPSXC5GDFXHA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.301379636 (12h3m22.0454365s) [noise=true , address=104.36.82.47:4546 , piece_id=5EQEI4X4MDF5IG3MZVTUBKDTFQEEKGZCCD7BGYFZXGK33D6D4MKQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.315130147 (12h3m22.05918701s) [noise=true , address=104.36.82.62:4410 , piece_id=JRUOOW3THAOPUO33WWSB6H3L2XDVXF5PGTUESAPTW6FKQO5FAVGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.351162615 (12h3m22.095219465s) [noise=true , address=104.36.82.55:4212 , piece_id=NXZ7HXDNRAJEWQPPKO4OUT3YSKQTR6EBD5BWIDUJHJOZLPSZKWBQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.383164149 (12h3m22.127221015s) [noise=true , address=104.36.82.35:4613 , piece_id=3R57UW3QIV7MWCRXNA7PBHZ7MMCJ4HJH74DRSNQ332MDY6CACSSA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.418050578 (12h3m22.162107442s) [noise=true , address=45.91.116.14:4205 , piece_id=34BYPMC76LGPOLPON2KXYCR7QXBEER6ONBH3ACWUGLK2FPE6CG4Q, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.451823428 (12h3m22.195880292s) [noise=true , address=104.36.82.44:4893 , piece_id=C6QOXHD34FQTQ32XNSXYJ7CTTGGQ5D5F2TDYMPNFPINSRT6BGL5A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.467192643 (12h3m22.211249504s) [noise=true , address=104.36.82.53:4128 , piece_id=NOOXQYQBBBYNFHL6R7T4HWLM4WGKQGZJ4FIU5GLXAHOCL4EJLM7A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.495563998 (12h3m22.239620847s) [noise=true , address=104.36.82.43:4857 , piece_id=VORSEMUSXKPTPW7VTIR4MQXNNNDAL3C5K4ECOMG73UHJPA7H6FAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.580244990 (12h3m22.324301836s) [noise=true , address=104.36.82.38:4714 , piece_id=YCKBGIMXXMTHKPAPETDOCDTILF5ZL5YFARTG5LTJP5CBMYN4XU3A, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.647231825 (12h3m22.391288672s) [noise=true , address=104.36.82.4:4811 , piece_id=DMH3FIBIW3K7GATC6NNFQJTHGCWDYLFTTHCSBNH3CK6CGZPFYBQA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.702946639 (12h3m22.447003485s) [noise=true , address=104.36.82.18:4384 , piece_id=ABREZ2F5MD2RMD2MXJKQOL5EOUABHYMNIANY5FKXXGOFQYFEDBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.748375718 (12h3m22.492432579s) [noise=true , address=104.36.82.59:4315 , piece_id=V4AKK2CP2XSGLVQUGA5HVGST6GWSMFGO7WVIXF7HQHGGJETVZDMA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801483011 (12h3m22.545539858s) [noise=true , address=79.127.205.233:4624 , piece_id=HJG5GPCPAWMHY62G3XI747GK5KE4QL57VGKJQQLX2DFRUKOZCUAQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: <nil> Aug 7 21:43:00.801509099 (12h3m22.545565964s) [seg_pos={0 0}, upload=12439]: Segment reached optimal threshold of 14 pieces. Aug 7 21:43:00.801678905 (12h3m22.545735756s) [noise=true , address=104.36.82.56:4230 , piece_id=ONDPLOX4QWPBZV52GXL7VSJGESTCGYASBMFUYEDRFKIREQO4RAIA, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802028506 (12h3m22.546085367s) [noise=true , address=64.49.13.138:4488 , piece_id=F4TDZWZ47Y5OCIWJCWIWJC57TGPQFCZ53TYX5IZZ2P6CTGTW6THQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802168197 (12h3m22.546225051s) [noise=true , address=104.36.82.9:4034 , piece_id=CDDRSJC4LD42VBSL7ARTJEAATGBXHOAEWYCZJF3V2KLAJHTTHBGQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY): protocol: expected piece hash; context canceled; EOF Aug 7 21:43:00.802240477 (12h3m22.546297327s) [noise=true , address=104.36.82.37:4654 , piece_id=KZ2LUEI6FVY2X7JFPPAPGFNJUMGPTK76B374L2IKCKGHXAMZ4VFQ, seg_pos={0 0}, upload=12439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: read tcp 139.59.144.22:42920->104.36.82.37:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:43:00.802254494 (12h3m22.546311343s) [seg_pos={0 0}, upload=12439]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:43:00.802379259 (12h3m22.546436112s) [upload=12439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:43:00.802386197 (12h3m22.546443069s) [upload=12439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:02.245820404 (12h3m23.98987728s) [upload=12439]: Done uploading segments... Aug 7 21:43:02.245849129 (12h3m23.98990599s) [upload=12439]: Upload finished. err: <nil> Duration: 4.166760951s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 3.655199588s FAIL (2025-08-07 21:43:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.655224911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 30.001446856s FAIL (2025-08-07 21:42:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001473523s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 37 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 16.288249129s FAIL (2025-08-07 21:42:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 16.288273969s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 23.276979187s FAIL (2025-08-07 21:42:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.277003075s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)