Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XXX.XXX............................] duration: 2.412163078s FAIL (2025-07-11 22:29:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.412193039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................XX..XX..............................] duration: 2.503490475s FAIL (2025-07-11 22:28:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.503518885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................XXX.XXXXX...........................] duration: 2.702919489s FAIL (2025-07-11 22:28:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.703020912s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XXX.XXX...........................] duration: 2.513733652s FAIL (2025-07-11 22:28:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.513766643s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX..XX.............................] duration: 2.545187309s FAIL (2025-07-11 22:27:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.545216805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................XXX.XXXXX..........................] duration: 2.684570495s FAIL (2025-07-11 22:27:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.684601164s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XXX.XXX..........................] duration: 2.344071817s FAIL (2025-07-11 22:27:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.344123989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX..XX............................] duration: 2.43466841s FAIL (2025-07-11 22:27:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.434697841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XXX.XXX.........................] duration: 2.705745842s FAIL (2025-07-11 22:26:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.705780101s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................XXX.XXXXX.........................] duration: 2.641004s FAIL (2025-07-11 22:26:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.641025574s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................XX..XX...........................] duration: 2.73817307s FAIL (2025-07-11 22:26:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.738206542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX.XXX........................] duration: 2.298297082s FAIL (2025-07-11 22:26:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.298329788s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................XXX.XXXXX........................] duration: 2.768560474s FAIL (2025-07-11 22:25:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.768588135s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................XX..XX..........................] duration: 2.662346518s FAIL (2025-07-11 22:25:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.662377742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX..XX.........................] duration: 2.781831061s FAIL (2025-07-11 22:25:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.781864679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX.XXX.......................] duration: 2.370833806s FAIL (2025-07-11 22:25:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.370872594s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................XXX.XXXXX.......................] duration: 2.637858303s FAIL (2025-07-11 22:24:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.637886198s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX..XX........................] duration: 2.706414137s FAIL (2025-07-11 22:24:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.706452649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXX.XXX......................] duration: 2.409629869s FAIL (2025-07-11 22:24:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.409662546s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................XXX.XXXXX......................] duration: 2.732484891s FAIL (2025-07-11 22:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.732513171s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX..XX.......................] duration: 2.997995284s FAIL (2025-07-11 22:23:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.99802052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................XXX.XXXXX.....................] duration: 2.687044516s FAIL (2025-07-11 22:23:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.687079994s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXX.XXX.....................] duration: 2.95244899s FAIL (2025-07-11 22:23:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.952491635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XXX.XXX....................] duration: 2.512925964s FAIL (2025-07-11 22:22:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.512976249s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............................XXX.XXXXX....................] duration: 2.844129035s FAIL (2025-07-11 22:22:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.844183617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................................XX..XX......................] duration: 2.855527077s FAIL (2025-07-11 22:22:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.855561144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XXX.XXX...................] duration: 2.295321879s FAIL (2025-07-11 22:22:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.295354117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX..XX.....................] duration: 3.010996686s FAIL (2025-07-11 22:22:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.011029737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................................XXX.XXXXX...................] duration: 2.707620432s FAIL (2025-07-11 22:22:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.707650188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XXX.XXX..................] duration: 2.291683473s FAIL (2025-07-11 22:21:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.29171266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX..XX....................] duration: 2.581445447s FAIL (2025-07-11 22:21:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.581474139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................XXX.XXXXX..................] duration: 2.856659629s FAIL (2025-07-11 22:20:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.856685624s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX..XX...................] duration: 2.913154739s FAIL (2025-07-11 22:20:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.913186907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................XXX.XXX.................] duration: 2.536010739s FAIL (2025-07-11 22:20:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.536051931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................................XX..XX..................] duration: 2.481260751s FAIL (2025-07-11 22:20:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.481317591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................................XXX.XXXXX.................] duration: 2.571143052s FAIL (2025-07-11 22:19:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.57117628s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XXX.XXX................] duration: 2.281239613s FAIL (2025-07-11 22:19:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.281267894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX..XX.................] duration: 2.682034855s FAIL (2025-07-11 22:19:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.682067485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................................XXX.XXXXX................] duration: 2.645688233s FAIL (2025-07-11 22:19:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.645721148s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................XXX.XXXXX...............] duration: 2.687638543s FAIL (2025-07-11 22:19:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.687668189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX..XX................] duration: 2.641182659s FAIL (2025-07-11 22:18:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.641233677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XXX.XXX...............] duration: 2.517970289s FAIL (2025-07-11 22:18:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.51800549s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................................XXX.XXXXX..............] duration: 2.593210639s FAIL (2025-07-11 22:18:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.593235926s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XXX.XXX..............] duration: 2.354888887s FAIL (2025-07-11 22:18:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.354922423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX..XX...............] duration: 2.622536169s FAIL (2025-07-11 22:18:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.622569938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................XX..XX..............] duration: 2.624005898s FAIL (2025-07-11 22:17:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.624038258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................................XXX.XXXXX.............] duration: 2.655313909s FAIL (2025-07-11 22:17:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.655334049s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXX.XXX.............] duration: 2.288902904s FAIL (2025-07-11 22:17:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 05.834196117 (444h56m5.717609755s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834843197 (444h56m5.718256835s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.834982650 (444h56m5.718396288s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835872519 (444h56m5.719286158s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.835927919 (444h56m5.719341551s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836802263 (444h56m5.7202159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.836874595 (444h56m5.720288233s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837685342 (444h56m5.721098983s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.837838250 (444h56m5.721251887s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838450447 (444h56m5.721864083s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.838843870 (444h56m5.722257507s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.839287996 (444h56m5.722701631s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840402981 (444h56m5.723816613s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.840797623 (444h56m5.724211252s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841281296 (444h56m5.724694929s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:05.841695089 (444h56m5.725108718s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Uploading piece... Jul 11 22:07:06.462006301 (444h56m6.345419934s) [upload=355799]: Next returned nil segment Jul 11 22:07:06.462031836 (444h56m6.345445474s) [upload=355799]: Waiting for error group managing segments... Jul 11 22:07:06.598768003 (444h56m6.48218162s) [noise=true , address=45.91.116.11:4119 , piece_id=PVHH7W43BX6YDWSTCDJIRAPMQO7NTIO5KLZC6KKK3TQCKOAICRNA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.602310096 (444h56m6.485723719s) [noise=true , address=45.91.116.8:4047 , piece_id=MVP45GE7QJ7HDS5AGQKIZF5GLSCARH2KWX4UNCC3UBITYQS7S2OQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:06.629301214 (444h56m6.512714826s) [noise=true , address=45.91.116.7:4010 , piece_id=CJUVJNAZCYXL7XDTRDJ33UMQWZ7H3MHDRTQFIRYNJQIVUNQFOQZQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.150234933 (444h56m8.033648546s) [noise=true , address=77.90.31.10:4517 , piece_id=XEBB7CHHWC2OXFHWGVBEZ2K2USUXGCU3JDUO5USFFGO6TWLBEY7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:08.496869058 (444h56m8.380282674s) [noise=true , address=77.90.31.12:4590 , piece_id=MWFJ54TQUCJMIEYVNAIKTHDTZKEODZYEA3OG5TVAP3BCIEVVOLVA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:09.251099401 (444h56m9.13451302s) [noise=true , address=77.90.31.11:4555 , piece_id=DPK4OJKTRZ6D7YFE36BBQI7BTB3FTKUGBAI7B4H3FKFCMJLPI3GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: <nil> Jul 11 22:07:35.448788537 (444h56m35.332202156s) [noise=true , address=104.36.82.12:4151 , piece_id=OWRDAGMSSWCKNQDSDTLX5RUEKUCQLLBWNWJGEHYIRN3XISUBNOBA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:54572->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448869305 (444h56m35.332282922s) [noise=true , address=104.36.82.11:4114 , piece_id=X6VGOMGCLHMJADYHPJRW3HJW2WTRDDAPVW42676EY6ONHVR6I4GA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:42960->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448926785 (444h56m35.332340401s) [noise=true , address=104.36.82.8:4972 , piece_id=H422QR7RSKFETN3EXKOGH4DCU3LYTVPJ4MCCOEEAPGNJ32TDGPYQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 139.59.144.22:33578->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.448980454 (444h56m35.332394072s) [noise=true , address=104.36.82.13:4186 , piece_id=KXX7AKAIUM3TKGJ7KBTAWOCJIWCRN5QYFLMFZNRD2K6MYSI6LC4Q, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 139.59.144.22:53804->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449012540 (444h56m35.332426159s) [noise=true , address=104.36.82.14:4222 , piece_id=LIDA4EYN65LVIYBXMQYZELR73ZJQBJF7HTPOARMYGEBHB74THGFA, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:58276->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449145824 (444h56m35.332559453s) [noise=true , address=104.36.82.9:4007 , piece_id=3AFAOFKNOT5RGIZCP7YEUKL4P4SSTN27QJQ3QNEDJPPBP6OJXOJQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:46048->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449231791 (444h56m35.332645408s) [noise=true , address=104.36.82.6:4876 , piece_id=ORBK5BUTLAN2QLDPAXNPBPFZB243AOYZTFHS2JXKVXJX544VJX7A, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:50862->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449288617 (444h56m35.332702234s) [noise=true , address=104.36.82.6:4874 , piece_id=3OVKEFBZHC4O6VV5YMBOZ4P4EBJM4E3YVB23E6DARRRISEAIZ3NQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 139.59.144.22:58020->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449303227 (444h56m35.332716868s) [noise=true , address=104.36.82.11:4113 , piece_id=U7QFQEMAAIAPLPEUD446QIC3LLU6VSWCXUYB6ZF66IPT5JS73WMQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:45960->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449358869 (444h56m35.332772487s) [noise=true , address=104.36.82.4:4802 , piece_id=C5ZYYVTDYZ5RVSUZ2TEVMOQT27QKLFODDNWVUJSVWNXK55HT2RPQ, seg_pos={0 0}, upload=355799]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 139.59.144.22:36524->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:35.449385152 (444h56m35.332798781s) [seg_pos={0 0}, upload=355799]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 11 22:07:35.559959616 (444h56m35.443373245s) [upload=355799]: Done uploading segments... Jul 11 22:07:35.559989423 (444h56m35.443403049s) [upload=355799]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.288935962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XX..XX.............] duration: 2.849358785s FAIL (2025-07-11 22:16:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 11 22:05:55.321813589 (444h54m55.205227205s) [upload=355785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:55.321818853 (444h54m55.20523248s) [upload=355785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:55.518997859 (444h54m55.402411489s) [seg_pos={0 0}, upload=355785]: Begin upload segment... Jul 11 22:05:55.519056590 (444h54m55.402470217s) [seg_pos={0 0}, upload=355785]: Done begin upload segment. Jul 11 22:05:55.519084972 (444h54m55.402498604s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519101144 (444h54m55.402514779s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519505827 (444h54m55.40291946s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519512227 (444h54m55.402925846s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519800709 (444h54m55.403214338s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.519822072 (444h54m55.403235701s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520126680 (444h54m55.403540312s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520132570 (444h54m55.403546202s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520439936 (444h54m55.403853566s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.520760592 (444h54m55.404174221s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521209831 (444h54m55.404623462s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521511418 (444h54m55.404925045s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.521813719 (444h54m55.405227352s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522154099 (444h54m55.405567732s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522438022 (444h54m55.405851648s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.522736799 (444h54m55.406150429s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.523027776 (444h54m55.406441402s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:55.524220461 (444h54m55.407634093s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Uploading piece... Jul 11 22:05:56.235898282 (444h54m56.119311907s) [upload=355785]: Next returned nil segment Jul 11 22:05:56.235916892 (444h54m56.119330525s) [upload=355785]: Waiting for error group managing segments... Jul 11 22:05:56.412884684 (444h54m56.296298314s) [noise=true , address=79.127.220.98:4431 , piece_id=ISLINB5HRRHMGSW74U4FNH7KP5L3IBJYVWQ4AWGSKUIMC3BOZIKA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.438943831 (444h54m56.322357474s) [noise=true , address=79.127.201.215:4336 , piece_id=HJXWRYIAWYVNUQQUXUN2GQETA3FI7N64JUNJFQ6Y5WR5HI2IGIUA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.460340320 (444h54m56.343753939s) [noise=true , address=79.127.220.101:4064 , piece_id=OQ6SPZNX2HK6HQBHW34VP4UK4HEJYRTBYKOFBRDEA3ON7ET4CN6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:56.780127785 (444h54m56.663541405s) [noise=true , address=64.49.13.27:4965 , piece_id=2RMLKEWRN2VUMCKLPZQYJBYHISY4FG6LWPOLJNK3DDQTWN7OIMQA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.130733263 (444h54m57.01414688s) [noise=true , address=77.90.31.12:4597 , piece_id=LQZFF56AWIDFQHJKQVJ2G5HVFFDPTGVKFVASQKDTASFZNKOLXJGA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:57.194331049 (444h54m57.077744665s) [noise=true , address=77.90.31.16:4768 , piece_id=GY7WDXBTBPNULKIMV7KNWJDQ4YL6HWVOQ7ZX7NRSYO6WXQXM2HJA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.269207970 (444h54m58.152621586s) [noise=true , address=77.90.31.15:4712 , piece_id=252M34NLO35YFQTYC4C7LSIRGSU2MI7VQQGR35EF24GNBX7WAI6A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:05:58.676775084 (444h54m58.560188697s) [noise=true , address=77.90.31.17:4791 , piece_id=P7BWLLYPSLTDZOIBKHSAN4Y2YHJRSCX4XI23W7L7Q7A4GTXJAW2A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:00.880812806 (444h55m0.764226436s) [noise=true , address=77.90.31.11:4562 , piece_id=XZEOCDLDVXX2IURENO6CLVJ6OLM62ZULCCDSH3POS45WY4ZN56LQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.885397124 (444h55m1.768810739s) [noise=true , address=77.90.31.11:4580 , piece_id=KPFBCF3NZXIRYABM3RIXNI4FSRF7S6VWMYTPR4QW6AVTRWA5CICQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:01.908450471 (444h55m1.79186409s) [noise=true , address=45.91.95.2:4035 , piece_id=RU7LOAUF77HCLG7WD6UIAPCBQY4WRS5Z4GZCBIYBPLLIRDNRMRXQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:02.920374484 (444h55m2.8037881s) [noise=true , address=45.91.95.5:4030 , piece_id=KX7ANCBUFRK2UQ4JSZ72FRRSXFJMFOO2JH4LIZBIKMDQ52S7MCTQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:06.990109655 (444h55m6.873523282s) [noise=true , address=45.91.95.3:4022 , piece_id=MIZNO3WDV64YO6EOQFWSPFLLXFLKHFVKRKLKUDQTMKAKQT3ZBOHQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: <nil> Jul 11 22:06:55.050519693 (444h55m54.93393332s) [noise=true , address=104.36.82.20:4463 , piece_id=3KYB27ICW3GAYHZIRQRI3RKI3KKT34IFDMGXN7CTMIBVFLPDI72A, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: write tcp 139.59.144.22:33618->104.36.82.20:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050607213 (444h55m54.934020832s) [noise=true , address=104.36.82.50:4031 , piece_id=2TJ6WQNZZIQBFFAHBXXCKMLRRUR2CGMNJC7UGIP5OMW5TZFNRW5Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 139.59.144.22:41868->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050629544 (444h55m54.934043166s) [noise=true , address=104.36.82.5:4861 , piece_id=W5WERDTIBDISS6SFK4NWSAD5SM4L5EUMTSGX5R3PH7EB3K7GGOLA, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 139.59.144.22:50224->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050646322 (444h55m54.934059943s) [noise=true , address=104.36.82.4:4824 , piece_id=E5J5EF6P2RFCSKMGY3CSPB4ZT2SIF2R7QXCJV6ZRIU2P6YUTAI6Q, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:104.36.82.4:4824): protocol: rpc: dial tcp 104.36.82.4:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050671506 (444h55m54.934085133s) [noise=true , address=104.36.82.22:4535 , piece_id=FWOYPFPCNITCDZL5TYIU4W6IJZ7PCEWDKHC3IBJ37TLVSPVC67QQ, seg_pos={0 0}, upload=355785]: Done uploading piece. err: ecclient: upload failed (node:1GEmVgDhSMyFLPY5qcc4gGqD9amUzhZ2iduyWusUSrVjLgBu1T, address:104.36.82.22:4535): protocol: write tcp 139.59.144.22:34212->104.36.82.22:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:55.050733238 (444h55m54.934146868s) [seg_pos={0 0}, upload=355785]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 11 22:06:55.167737334 (444h55m55.051150954s) [upload=355785]: Done uploading segments... Jul 11 22:06:55.167778274 (444h55m55.051191901s) [upload=355785]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.849392304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................................XXX.XXXXX............] duration: 2.611806193s FAIL (2025-07-11 22:16:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.61182772s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)