Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.899231734s FAIL (2025-07-05 20:25:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.899255591s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 405.674956ms FAIL (2025-07-05 20:24:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 405.772464ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.955684437s FAIL (2025-07-05 20:24:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.955709364s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 449.944395ms FAIL (2025-07-05 20:23:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 449.97468ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.697586559s FAIL (2025-07-05 20:23:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.697612209s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 430.36936ms FAIL (2025-07-05 20:22:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 430.404423ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.556926918s FAIL (2025-07-05 20:22:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.556957814s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 428.554935ms FAIL (2025-07-05 20:22:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 428.590137ms core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 456.121787ms FAIL (2025-07-05 20:21:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 456.155478ms core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 477.126413ms FAIL (2025-07-05 20:21:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 477.156937ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.090722986s FAIL (2025-07-05 20:21:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.090755754s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 451.523625ms FAIL (2025-07-05 20:20:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 451.55976ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.060262469s FAIL (2025-07-05 20:20:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.0602906s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 384.033567ms FAIL (2025-07-05 20:19:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 384.065091ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.771693095s FAIL (2025-07-05 20:19:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.771729301s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 410.748898ms FAIL (2025-07-05 20:18:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 410.78272ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.765611603s FAIL (2025-07-05 20:18:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.765640246s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 419.140058ms FAIL (2025-07-05 20:17:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 419.175847ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.806333573s FAIL (2025-07-05 20:17:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.806359032s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 484.449324ms FAIL (2025-07-05 20:17:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 484.481136ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.713481387s FAIL (2025-07-05 20:16:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.713507439s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 418.050355ms FAIL (2025-07-05 20:16:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 418.083628ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.886362586s FAIL (2025-07-05 20:16:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.886388153s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 491.563345ms FAIL (2025-07-05 20:16:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 491.660738ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.821600023s FAIL (2025-07-05 20:15:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.821641933s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 517.887046ms FAIL (2025-07-05 20:15:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 517.923657ms core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 402.071038ms FAIL (2025-07-05 20:14:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 402.108708ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.770027021s FAIL (2025-07-05 20:13:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.770052888s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.489200208s FAIL (2025-07-05 20:12:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.489229965s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 442.761188ms FAIL (2025-07-05 20:12:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 442.796798ms core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 398.205549ms FAIL (2025-07-05 20:12:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 398.23551ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.712134903s FAIL (2025-07-05 20:11:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.712159537s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 461.017838ms FAIL (2025-07-05 20:11:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 461.056781ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.779395534s FAIL (2025-07-05 20:11:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.779437984s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.73106978s FAIL (2025-07-05 20:10:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.731097754s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 15.758626067s FAIL (2025-07-05 20:10:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 15.758654652s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 480.653078ms FAIL (2025-07-05 20:10:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 480.690217ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.792144714s FAIL (2025-07-05 20:09:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.792481854s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 431.032031ms FAIL (2025-07-05 20:09:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 431.064989ms core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 440.234595ms FAIL (2025-07-05 20:08:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 440.268348ms core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.971634395s FAIL (2025-07-05 20:08:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.971660498s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.560845238s FAIL (2025-07-05 20:08:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.560901841s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 452.391319ms FAIL (2025-07-05 20:08:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 452.414497ms core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.029741305s FAIL (2025-07-05 20:08:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== 492 (298h56m2.215210125s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Uploading piece... Jul 5 20:07:02.431396637 (298h56m2.314810268s) [upload=239112]: Next returned nil segment Jul 5 20:07:02.431421007 (298h56m2.314834638s) [upload=239112]: Waiting for error group managing segments... Jul 5 20:07:02.447095606 (298h56m2.330509226s) [noise=true , address=89.149.201.147:4019 , piece_id=MAUEKAJFTZNJMXF57JO2VBOV7L4ZZKILGR65GRDLBSV22K3R2TXA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.448042898 (298h56m2.331456515s) [noise=true , address=89.149.201.147:4024 , piece_id=OT6RRS4XBYINP4QB3RJ7VWPY2AD7GJPAQIW2ATM6ZELXOWA526RQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.449028099 (298h56m2.332441731s) [noise=true , address=89.149.201.147:4005 , piece_id=JQQG4Y45XA3XZMIAKWUPLGEIRBGVNHM5NCWRTULQTVPYXO6FO55Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.450682866 (298h56m2.334096485s) [noise=true , address=81.171.14.70:4034 , piece_id=IFDTTFWJPB636YM4NP6747UFLLZE4KMVLQYO7J4KVY4G6QWSZXSQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.451864269 (298h56m2.335277896s) [noise=true , address=81.171.14.70:4032 , piece_id=O3DKC6N4KXI7IUOVRSJSLP6TW6XDWZNES7BH3KZXAPNUYWC5O2DA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.456410299 (298h56m2.339823915s) [noise=true , address=81.171.14.70:4044 , piece_id=D3LDMEEASIEDNMZWVBYYAVZHRLNQIRGLRDS2LELUCSI52KASMJNQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.457434058 (298h56m2.340847673s) [noise=true , address=81.171.14.70:4027 , piece_id=HSWN54OPPTUAXJ7SNL2QXEEZK2HMSMJSTYWKHL5LBUN2YB4ZB6FQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.461640690 (298h56m2.345054326s) [noise=true , address=89.149.201.147:4023 , piece_id=A663SYQPHVRID3JBQG5BJL4F74AV33U4TYLDGJSXMQIZS5OEXJVQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.470937271 (298h56m2.354350905s) [noise=true , address=89.149.201.147:4011 , piece_id=KJAFCNL5N2D4OUPG3NEVOMWOIHTXG7GZQYXKV2DG2GHZ3GELVARA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.479272848 (298h56m2.362686482s) [noise=true , address=81.171.14.70:4028 , piece_id=VVC3RQ2YDSK7BNDCMLVHD4ZOCOA3LDU25UWUD3DZEEID3KSSNEEQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.480764798 (298h56m2.364178415s) [noise=true , address=81.171.14.70:4036 , piece_id=M4HHHX7L3PHNMJJBRF7KITDTOCWMLXMYY7EAB6UZJQZYXEU6AHZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.481062475 (298h56m2.364476092s) [noise=true , address=89.149.201.147:4013 , piece_id=TCB2SEVENWPWROT6UVTQNFBCZQXY2BNGCJJSL5ZVNDCGM6VQ4PCQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.486941056 (298h56m2.37035467s) [noise=true , address=89.149.201.147:4008 , piece_id=VHDOIVNIDASIUA7YC2OR6GCNWA7MSYOZZBGLJDKI4RBMNCBALF2Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.488291026 (298h56m2.371704665s) [noise=true , address=89.149.201.147:4015 , piece_id=NBRDDGC23DXKQNGOFZHBJ25I6QI7TZYUJ5XCENKCJOJO6CE2QFPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:07:02.524985461 (298h56m2.40839908s) [noise=true , address=89.149.201.147:4002 , piece_id=2LMWF6FTCFJFTMZQK2SK4KBRDDYVJTSXNKM5A6FK6KOCRPXDYRUA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: <nil> Jul 5 20:08:02.219008955 (298h57m2.102422574s) [noise=true , address=185.167.204.205:4036 , piece_id=6DQYOQ3IEEJEE442KTZ7R3JCLGHJJTSKG4AUZFP5I4G2UOPMLWPQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Mngc2aRWoo1ZQGktFfA4DauhKyEFcbRk6dg7nYa1jZUg2srMV, address:185.167.204.205:4036): protocol: write tcp 139.59.144.22:51574->185.167.204.205:4036: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219117336 (298h57m2.102530954s) [noise=true , address=185.167.204.205:4013 , piece_id=B3AU5MMLPEFO22IMVEQ3LVEJSA4PFUG6V4GJFE2R3WOMUHH3VGJQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1DJ9zLuqKQthmHgaSqovGQo8Sq4h8ymNV3vYc5XapvsxCVBrL5, address:185.167.204.205:4013): protocol: rpc: dial tcp 185.167.204.205:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219177020 (298h57m2.102590639s) [noise=true , address=185.167.204.205:4035 , piece_id=EO5BO6VXN6HXRZRQN5TAPTNG4CNXQZXFSNYLCQS3DWCQ53XXIYDA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219237059 (298h57m2.102650678s) [noise=true , address=185.167.204.206:4054 , piece_id=JBEJ4R5DXATRMFFRTD3C3ONJV2H2JA26ZF2HH7EHG5KJ3BYZ7KSA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12ExKH4MVQ3r3VHXmBuTWSGAsqp9DyxxsojedEu96rtn4Jwk17u, address:185.167.204.206:4054): protocol: rpc: dial tcp 185.167.204.206:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219293293 (298h57m2.102706912s) [noise=true , address=185.167.204.205:4002 , piece_id=FMBOS5TD46M5HJKAYM67NI6YUFAD3LQXKIVPTP5GJ4MMKDOX3LGA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219349725 (298h57m2.102763345s) [noise=true , address=185.167.204.206:4064 , piece_id=7ZSIBWN4TK6GOGXX5EE5MGCB5OYZWUCP6UUJO6W46IDDBH5PJM4A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.219382140 (298h57m2.102795759s) [noise=true , address=185.167.204.206:4052 , piece_id=QTUGBOXPYWCVTZBM7WPLYIIQU62KKJZU76FPRRB2Y4LJVNGPTNZQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220015537 (298h57m2.103429157s) [noise=true , address=185.167.204.206:4065 , piece_id=IHI3OAB4YGQVXPUE6V72Y5CCRJNARF2GXAKJPQAGX6A7VIQWCS5Q, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220073820 (298h57m2.103487439s) [noise=true , address=185.167.204.205:4006 , piece_id=UBVRDRVJ4A3UN6XUNR3CEKWQFM3OKPCYVWDUTGSQUG47BYW32FQQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: rpc: dial tcp 185.167.204.205:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220132743 (298h57m2.103546363s) [noise=true , address=185.167.204.205:4011 , piece_id=PX542Q2X6YQCEHSJJCGTQ67VPNHQFUVTGBY2H6A2K2N5LNX6MIOQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12CiRxadZgQLgJ41bv86hSTEHLv2CzXC4jz9Q3r5dLEX6doYoZ3, address:185.167.204.205:4011): protocol: rpc: dial tcp 185.167.204.205:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220185981 (298h57m2.1035996s) [noise=true , address=185.167.204.205:4024 , piece_id=FVAR6VVKANNWC4EUMBDI2GHMHBIFQZWO7QAIKJXJYQUVMNS5AVHQ, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12Wa3aDy81HhCKQiRo9BeZGYZqcVJWbLpN4TxdZaMHuitPSTxLV, address:185.167.204.205:4024): protocol: write tcp 139.59.144.22:35118->185.167.204.205:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.220252973 (298h57m2.103666593s) [noise=true , address=185.167.204.205:4030 , piece_id=T6XQJHO4KQYI7WBXVDXQ22FMMUWK3EJTCFNS33V4J7NAGVDAAL7A, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 5 20:08:02.225370789 (298h57m2.108784408s) [noise=true , address=185.167.204.205:4009 , piece_id=OMCAI3O25625Q76QGQ2QH6YND3QFWSCUVBC5M66G63GOWX6HZ5NA, seg_pos={0 0}, upload=239112]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 5 20:08:02.225430620 (298h57m2.108844238s) [seg_pos={0 0}, upload=239112]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> Jul 5 20:08:02.245478781 (298h57m2.128892402s) [upload=239112]: Done uploading segments... Jul 5 20:08:02.245512127 (298h57m2.128925759s) [upload=239112]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 1m0.029775898s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.7018206s FAIL (2025-07-05 20:07:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.701845118s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.77522949s FAIL (2025-07-05 20:06:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.775258143s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.222940102s FAIL (2025-07-05 20:06:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.222971944s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.525105401s FAIL (2025-07-05 20:05:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.525135976s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.820769732s FAIL (2025-07-05 20:05:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.820804081s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.753793667s FAIL (2025-07-05 20:04:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.753849447s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)