Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.643175844s
FAIL (2025-10-21 21:42:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.643201862s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.301001431s
FAIL (2025-10-21 21:42:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.301050696s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.876326699s
FAIL (2025-10-21 21:42:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.876357133s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
1.451163816s
FAIL (2025-10-21 21:42:10 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.451203636s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.551731317s
FAIL (2025-10-21 21:42:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.551770693s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.381783651s
FAIL (2025-10-21 21:41:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.381821893s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.578124835s
FAIL (2025-10-21 21:41:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.578151819s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
1.492596695s
FAIL (2025-10-21 21:40:56 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.492629213s
core
uplink
ap1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.719056183s
FAIL (2025-10-21 21:40:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.71909121s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.264260013s
FAIL (2025-10-21 21:40:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.264308223s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.46043479s
FAIL (2025-10-21 21:40:17 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.460465944s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.430889757s
FAIL (2025-10-21 21:40:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.430943052s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.314596934s
FAIL (2025-10-21 21:39:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.314628973s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.824748978s
FAIL (2025-10-21 21:39:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.824828402s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.52345765s
FAIL (2025-10-21 21:39:21 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.523482741s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000732882s
FAIL (2025-10-21 21:39:19 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.000758598s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.260479617s
FAIL (2025-10-21 21:38:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.260510716s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.496270795s
FAIL (2025-10-21 21:38:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.497576671s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.32909122s
FAIL (2025-10-21 21:38:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.329166017s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.526727737s
FAIL (2025-10-21 21:38:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.526757627s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.655324777s
FAIL (2025-10-21 21:37:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.655505175s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.684756749s
FAIL (2025-10-21 21:37:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.684789613s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.246598411s
FAIL (2025-10-21 21:36:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.246629291s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.677197452s
FAIL (2025-10-21 21:36:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.677225476s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.287589558s
FAIL (2025-10-21 21:35:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.287624477s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.538333914s
FAIL (2025-10-21 21:35:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.538358922s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.618953743s
FAIL (2025-10-21 21:34:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.6189856s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.445531519s
FAIL (2025-10-21 21:34:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.445561262s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.67205175s
FAIL (2025-10-21 21:33:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.672081305s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
2.387753329s
FAIL (2025-10-21 21:33:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.387791712s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.669189296s
FAIL (2025-10-21 21:33:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.669218617s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
2.380882675s
FAIL (2025-10-21 21:32:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.380912453s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
2.786307568s
FAIL (2025-10-21 21:32:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.787295934s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.421252635s
FAIL (2025-10-21 21:32:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.421301203s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
2.610957056s
FAIL (2025-10-21 21:31:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.610985142s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.399069258s
FAIL (2025-10-21 21:31:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.399091904s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.6497577s
FAIL (2025-10-21 21:31:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.649787017s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.738070635s
FAIL (2025-10-21 21:30:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.738098649s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.295210383s
FAIL (2025-10-21 21:30:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.295243109s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.29714097s
FAIL (2025-10-21 21:29:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.297210829s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.851809408s
FAIL (2025-10-21 21:29:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.851838155s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.618205353s
FAIL (2025-10-21 21:29:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.618284739s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.32998449s
FAIL (2025-10-21 21:29:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.330017097s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.517601411s
FAIL (2025-10-21 21:28:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.517634532s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.332558213s
FAIL (2025-10-21 21:28:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.332590022s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.394240223s
FAIL (2025-10-21 21:27:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.394285741s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.680751281s
FAIL (2025-10-21 21:27:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.680823671s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.449940379s
FAIL (2025-10-21 21:26:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.449975966s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.529518951s
FAIL (2025-10-21 21:26:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.529549895s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.11921735s
FAIL (2025-10-21 21:26:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
79h55m56.299224296s) [seg_pos={0 0}, upload=598259]: Begin upload segment...
Oct 21 21:26:08.495884845 (679h55m56.299349445s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.496263329 (679h55m56.299727932s) [seg_pos={0 0}, upload=598259]: Done begin upload segment.
Oct 21 21:26:08.496307238 (679h55m56.299771843s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497412950 (679h55m56.300877554s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497681783 (679h55m56.301146387s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.497966420 (679h55m56.301431033s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498322244 (679h55m56.301786847s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498446002 (679h55m56.30191061s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.498944412 (679h55m56.302409014s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.499504528 (679h55m56.302969129s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.500710908 (679h55m56.304175504s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.501945299 (679h55m56.305409901s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.503349707 (679h55m56.306814307s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.505306389 (679h55m56.308770991s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.507500478 (679h55m56.310965083s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.510112475 (679h55m56.313577077s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:08.515434693 (679h55m56.318899295s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Uploading piece...
Oct 21 21:26:09.111799758 (679h55m56.915264349s) [upload=598259]: Next returned nil segment
Oct 21 21:26:09.111825994 (679h55m56.915290595s) [upload=598259]: Waiting for error group managing segments...
Oct 21 21:26:09.250630272 (679h55m57.054094858s) [noise=true , address=145.79.227.13:4804 , piece_id=EX4QYQWMJ5JL6Z4VGJZ7Q6JQQ6MYV3A6LWXVWVIV4AN6DWAVKMBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.250861642 (679h55m57.054326226s) [noise=true , address=145.79.227.17:4972 , piece_id=EH2QLQ2QFSNWEAZKHSQKCF3MZM4H5MITH762QQWND5P2IZKGLJRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.251085636 (679h55m57.054550219s) [noise=true , address=145.79.227.11:4730 , piece_id=QXCK5XS5VR33K2PXWFHZQEPZZ5L7U4USQJAT5KDRTSHNOUZU4NWQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.252043316 (679h55m57.055507896s) [noise=true , address=145.79.227.8:4329 , piece_id=75ANEH4G7RJ6EZPAOF2E2TUYLT5M6PFWFFHP6WGXIMDGZXBLMH3A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.254746616 (679h55m57.058211209s) [noise=true , address=145.79.227.16:4912 , piece_id=2HOPDVQNK6YO5ZKLTOIYKZLJEUKOXJJAAJE2EBSVOEQPW5IVWTQQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:09.273098197 (679h55m57.076562793s) [noise=true , address=145.79.227.6:4258 , piece_id=26W7SRPXZQS5DQGHV5SUJGCCG4FF4NB6RGI5SEFKFOWLST4HTJVQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:10.629898811 (679h55m58.433363394s) [noise=true , address=77.90.31.18:4807 , piece_id=E3NNAB5A5LMAB7LOUK7DMUZTGOLZ5E4QFTWAUTJKINKEFPBRWP6A, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:16.420197867 (679h56m4.223662448s) [noise=true , address=77.90.31.13:4628 , piece_id=HMLMTQAJEZ54JBEUTONZYF6OCF2WPDR7QFPDB7IAUFEPZZZLIPRQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:27.762835686 (679h56m15.566300265s) [noise=true , address=45.91.116.4:4940 , piece_id=PXNWSSGMV6TXVG6U6HIULKGCGUZCYSDZKYL2AK45IGJPVYRCOCBQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: <nil>
Oct 21 21:26:38.083181867 (679h56m25.88664645s) [noise=true , address=45.91.116.4:4937 , piece_id=YA7STPE6YXXBWD7HENDC6LE2KXQLO3V577NDDN3GZMFRLSVSA5UQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47400->45.91.116.4:4937: use of closed network connection
Oct 21 21:26:38.083254545 (679h56m25.886719125s) [noise=true , address=77.90.31.14:4662 , piece_id=BF6DJSTH4UVD5Z3VDKWCBRGPQS6AW2FFFKGT4WECLSUJRHBLTOXA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: expected piece hash; context deadline exceeded; EOF
Oct 21 21:26:38.083274067 (679h56m25.886738651s) [noise=true , address=77.90.31.15:4697 , piece_id=3QIPJV6JASUPK3WDFEXXP2MWPETPFEVHC4PSFZO6PGNQEPTJ42YA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49604->77.90.31.15:4697: use of closed network connection
Oct 21 21:26:38.083339462 (679h56m25.886804047s) [noise=true , address=77.90.31.16:4734 , piece_id=KAZWNLJG2UJPMRMSZVORYBPEOU3KBMJ6H7BFJS7SBTIS6YLBBMMQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:49426->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 21 21:26:38.083379127 (679h56m25.88684371s) [noise=true , address=77.90.31.12:4592 , piece_id=52HQLMW2E54Z7SMGN2JQVNC2YDE6NLHLUBFID72MWF7NCFDBGFUA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39984->77.90.31.12:4592: use of closed network connection
Oct 21 21:26:38.083453729 (679h56m25.886918325s) [noise=true , address=77.90.31.11:4553 , piece_id=T2PQTDOOPWD75R3SR5MW6YIVDJQMYOETLOWMY2IPDBZEQNLM4ZTQ, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59090->77.90.31.11:4553: use of closed network connection
Oct 21 21:26:38.083519269 (679h56m25.886983854s) [noise=true , address=77.90.31.10:4517 , piece_id=T2OQ4ZUGGGHRZDQRMKTNEFNF4BOEL5777DURFRUVIJOU5CHL3ACA, seg_pos={0 0}, upload=598259]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37548->77.90.31.10:4517: use of closed network connection
Oct 21 21:26:38.083705711 (679h56m25.887170399s) [seg_pos={0 0}, upload=598259]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 21 21:26:38.199686825 (679h56m26.003151424s) [upload=598259]: Done uploading segments...
Oct 21 21:26:38.199773196 (679h56m26.0032378s) [upload=598259]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.119278046s
core
uplink
us-select-1
|