Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.720255649s FAIL (2025-10-21 22:09:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.720289603s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.606700716s FAIL (2025-10-21 22:08:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.606729755s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.714998456s FAIL (2025-10-21 22:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.715070429s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.570786437s FAIL (2025-10-21 22:07:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.570847549s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.620794214s FAIL (2025-10-21 22:07:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.620845978s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.297107967s FAIL (2025-10-21 22:07:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.297138398s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.745595804s FAIL (2025-10-21 22:07:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.745635632s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.767359174s FAIL (2025-10-21 22:07:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.767441469s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.401763936s FAIL (2025-10-21 22:06:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.401798811s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.651896326s FAIL (2025-10-21 22:06:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.651932041s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.837383608s FAIL (2025-10-21 22:06:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.83748545s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.387051665s FAIL (2025-10-21 22:06:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.387093526s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.639726437s FAIL (2025-10-21 22:05:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.639766585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.365910469s FAIL (2025-10-21 22:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.365962412s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.661373802s FAIL (2025-10-21 22:05:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.661398647s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.668042849s FAIL (2025-10-21 22:05:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.668070677s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.831308193s FAIL (2025-10-21 22:05:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.831358042s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.500139396s FAIL (2025-10-21 22:05:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.500173127s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.653472689s FAIL (2025-10-21 22:04:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.653507511s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.361921549s FAIL (2025-10-21 22:04:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.361955621s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.513968224s FAIL (2025-10-21 22:04:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.514402085s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.407998123s FAIL (2025-10-21 22:04:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.408036056s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.609299914s FAIL (2025-10-21 22:04:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.609342492s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.647023835s FAIL (2025-10-21 22:03:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.647052084s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.374666533s FAIL (2025-10-21 22:03:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.374750913s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.490185039s FAIL (2025-10-21 22:03:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.490244089s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.710632718s FAIL (2025-10-21 22:02:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.710664006s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.455498312s FAIL (2025-10-21 22:02:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.455530736s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.513128594s FAIL (2025-10-21 22:02:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.51319028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.442695525s FAIL (2025-10-21 22:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.44274711s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.783856494s FAIL (2025-10-21 22:02:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.783897522s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.735331334s FAIL (2025-10-21 22:01:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.735359727s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.576740088s FAIL (2025-10-21 22:01:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.576789599s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.342378041s FAIL (2025-10-21 22:01:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.342412638s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.649268033s FAIL (2025-10-21 22:01:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.649295088s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.425419178s FAIL (2025-10-21 22:01:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.425453668s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.793217296s FAIL (2025-10-21 22:00:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.793241965s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.764529232s FAIL (2025-10-21 22:00:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.764569789s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.436670753s FAIL (2025-10-21 22:00:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.436704842s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.638494719s FAIL (2025-10-21 21:59:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.638531078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.492289138s FAIL (2025-10-21 21:59:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.492330794s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.671786508s FAIL (2025-10-21 21:59:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.671810757s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.423651736s FAIL (2025-10-21 21:59:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.423917826s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.413054258s FAIL (2025-10-21 21:59:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.413097336s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.708102994s FAIL (2025-10-21 21:58:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.708130986s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.623099296s FAIL (2025-10-21 21:58:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.623125122s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.715498355s FAIL (2025-10-21 21:58:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.715546488s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.356130908s FAIL (2025-10-21 21:58:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment... Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment. Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece... Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments... Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil> Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments... Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.356162137s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.589959517s FAIL (2025-10-21 21:57:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.58999217s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.622270936s FAIL (2025-10-21 21:57:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.622299336s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)