Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......XXX.XXXXX.............................................] duration: 2.828227346s FAIL (2025-07-11 22:40:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.828250881s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XXX.XXX............................................] duration: 2.445735851s FAIL (2025-07-11 22:40:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.445835064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XXX.XXX...........................................] duration: 2.381375484s FAIL (2025-07-11 22:40:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.38141037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........XX..XX..............................................] duration: 2.531406376s FAIL (2025-07-11 22:40:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.531437904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XXX.XXX..........................................] duration: 2.401299248s FAIL (2025-07-11 22:40:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.401351296s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......XXX.XXXXX............................................] duration: 2.695764193s FAIL (2025-07-11 22:40:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.69583647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........XX..XX.............................................] duration: 3.121773679s FAIL (2025-07-11 22:40:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.121810715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........XXX.XXXXX...........................................] duration: 2.598193887s FAIL (2025-07-11 22:39:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.598264825s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........XX..XX............................................] duration: 2.512104146s FAIL (2025-07-11 22:39:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.512148233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XXX.XXX.........................................] duration: 2.398585865s FAIL (2025-07-11 22:39:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.39862133s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........XXX.XXXXX..........................................] duration: 2.865416827s FAIL (2025-07-11 22:39:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.865443311s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........XXX.XXXXX.........................................] duration: 2.481483561s FAIL (2025-07-11 22:38:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.481505707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........XX..XX...........................................] duration: 2.448008181s FAIL (2025-07-11 22:38:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.44804259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XXX.XXX........................................] duration: 2.32972487s FAIL (2025-07-11 22:38:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.329780076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XXX.XXX.......................................] duration: 2.423215694s FAIL (2025-07-11 22:37:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.423265997s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........XXX.XXXXX........................................] duration: 2.68220088s FAIL (2025-07-11 22:37:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.682227948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............XX..XX..........................................] duration: 2.580654233s FAIL (2025-07-11 22:37:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.580682043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............XXX.XXXXX.......................................] duration: 2.82481532s FAIL (2025-07-11 22:37:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824878248s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............XX..XX.........................................] duration: 2.81671155s FAIL (2025-07-11 22:36:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.816745074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XXX.XXX......................................] duration: 2.420299405s FAIL (2025-07-11 22:36:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.420399408s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............XXX.XXXXX......................................] duration: 2.733238236s FAIL (2025-07-11 22:36:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.733333745s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............XX..XX........................................] duration: 2.719556841s FAIL (2025-07-11 22:36:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.719597243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............XXX.XXXXX.....................................] duration: 2.724337117s FAIL (2025-07-11 22:35:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.724358125s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................XXX.XXX.....................................] duration: 2.340731258s FAIL (2025-07-11 22:35:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.340781116s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............XXX.XXXXX....................................] duration: 2.82411204s FAIL (2025-07-11 22:35:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824139803s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XXX.XXX....................................] duration: 2.608893361s FAIL (2025-07-11 22:35:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.608929736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............XX..XX.......................................] duration: 2.622660782s FAIL (2025-07-11 22:35:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.622694732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................XXX.XXXXX...................................] duration: 2.742233343s FAIL (2025-07-11 22:34:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.742263227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XXX.XXX...................................] duration: 2.426175011s FAIL (2025-07-11 22:34:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.426206829s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................XXX.XXXXX..................................] duration: 2.761956065s FAIL (2025-07-11 22:34:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.761980077s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................XX..XX......................................] duration: 2.543247201s FAIL (2025-07-11 22:34:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.543279269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XXX.XXX..................................] duration: 2.347035855s FAIL (2025-07-11 22:33:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.347065463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................XX..XX.....................................] duration: 2.64688209s FAIL (2025-07-11 22:33:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.646922393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XXX.XXX.................................] duration: 2.565769594s FAIL (2025-07-11 22:33:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.565803156s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................XXX.XXXXX.................................] duration: 2.543295449s FAIL (2025-07-11 22:33:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.543396309s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XXX.XXX................................] duration: 3.242560849s FAIL (2025-07-11 22:32:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.242597274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................XX..XX....................................] duration: 6.323497878s FAIL (2025-07-11 22:32:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.323527141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................XX..XX...................................] duration: 5.974215466s FAIL (2025-07-11 22:32:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.974252274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................XXX.XXXXX................................] duration: 3.204674615s FAIL (2025-07-11 22:32:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.204732017s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XXX.XXX...............................] duration: 5.376091028s FAIL (2025-07-11 22:31:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.376122458s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................XXX.XXXXX...............................] duration: 2.661516276s FAIL (2025-07-11 22:31:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.661541375s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................XX..XX..................................] duration: 2.900438407s FAIL (2025-07-11 22:31:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.900470036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX..XX.................................] duration: 2.484092011s FAIL (2025-07-11 22:30:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.484124141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XXX.XXX..............................] duration: 2.507575527s FAIL (2025-07-11 22:30:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.507615678s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................XXX.XXXXX..............................] duration: 2.667035556s FAIL (2025-07-11 22:30:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.667631807s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................XX..XX................................] duration: 2.512269606s FAIL (2025-07-11 22:30:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.512299376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................XXX.XXXXX.............................] duration: 2.665411358s FAIL (2025-07-11 22:30:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.665440985s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XXX.XXX.............................] duration: 2.380876319s FAIL (2025-07-11 22:29:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.3810168s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................XXX.XXXXX............................] duration: 2.738750676s FAIL (2025-07-11 22:29:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.738775866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX..XX...............................] duration: 3.597614232s FAIL (2025-07-11 22:29:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.597647167s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)