Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....XX......X..................................XX......] duration: 2.499037262s FAIL (2025-09-22 03:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.499070879s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..........X.......X....X.X.......XX....X..X...X.......X..] duration: 2.908304385s FAIL (2025-09-22 03:28:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.908370083s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....XX......X..................................XX.....] duration: 2.399869463s FAIL (2025-09-22 03:28:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.399901106s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X..........X.......X....X.X.......XX....X..X...X.......X.] duration: 3.153640845s FAIL (2025-09-22 03:28:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.15366803s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X..........X.......X....X.X.......XX....X..X...X.......X] duration: 30.001082954s FAIL (2025-09-22 03:28:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001531056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....X.....XX......X..................................XX....] duration: 2.777851518s FAIL (2025-09-22 03:27:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.777880271s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X....X..........X.......X....X.X.......XX....X..X...X.......] duration: 2.673556414s FAIL (2025-09-22 03:26:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.673579202s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....X.....XX......X..................................XX...] duration: 2.418634998s FAIL (2025-09-22 03:26:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.418661552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X....X.....XX......X..................................XX..] duration: 2.44055344s FAIL (2025-09-22 03:26:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.440584823s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX....X..........X.......X....X.X.......XX....X..X...X......] duration: 2.845833055s FAIL (2025-09-22 03:26:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.845893256s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X....X.....XX......X..................................XX.] duration: 2.459198241s FAIL (2025-09-22 03:25:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.459292482s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX....X..........X.......X....X.X.......XX....X..X...X.....] duration: 2.760777936s FAIL (2025-09-22 03:25:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.760812212s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XX....X..........X.......X....X.X.......XX....X..X...X....] duration: 2.991064296s FAIL (2025-09-22 03:25:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.991102566s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.X....X.....XX......X..................................XX] duration: 30.103516568s FAIL (2025-09-22 03:25:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:24:49.723373695 (565h17m10.215806124s) [upload=497309]: Starting upload Sep 22 03:24:49.723442877 (565h17m10.215875318s) [upload=497309]: Done starting upload Sep 22 03:24:49.723559207 (565h17m10.215991652s) [upload=497309]: Uploading segments... Sep 22 03:24:49.723695446 (565h17m10.216127887s) [upload=497309]: Got next segment. Inline: false Sep 22 03:24:49.723736149 (565h17m10.216168577s) [upload=497309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:24:49.723740861 (565h17m10.216173289s) [upload=497309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:24:49.871368134 (565h17m10.363800581s) [seg_pos={0 0}, upload=497309]: Begin upload segment... Sep 22 03:24:49.871435764 (565h17m10.363868213s) [seg_pos={0 0}, upload=497309]: Done begin upload segment. Sep 22 03:24:49.871487100 (565h17m10.363919548s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.871946535 (565h17m10.364378979s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872276404 (565h17m10.364708849s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872412852 (565h17m10.364845295s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.872744232 (565h17m10.365176677s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873123905 (565h17m10.365556356s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873493977 (565h17m10.36592642s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.873817544 (565h17m10.366249987s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874416869 (565h17m10.366849332s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874450690 (565h17m10.366883132s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.874846463 (565h17m10.367278909s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875190183 (565h17m10.367622627s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875511139 (565h17m10.367943588s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.875869578 (565h17m10.368302022s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.876186798 (565h17m10.368619242s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:49.885649743 (565h17m10.378082172s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Uploading piece... Sep 22 03:24:50.485219822 (565h17m10.977652262s) [upload=497309]: Next returned nil segment Sep 22 03:24:50.485258559 (565h17m10.977691003s) [upload=497309]: Waiting for error group managing segments... Sep 22 03:24:50.626215248 (565h17m11.118647677s) [noise=true , address=45.91.116.9:4263 , piece_id=557UZ2CO2FRCEQ5VMKRN7A2MZ7MVWDC3T6U7FDTFESTLTVEIKJLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.626240268 (565h17m11.118672703s) [noise=true , address=45.91.116.11:4117 , piece_id=7QQGDENCX3VD6FA27UGWP5G2OY7TUZDPLWGB3KIIE522ROQPSGLA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.627351929 (565h17m11.119784361s) [noise=true , address=45.91.116.4:4939 , piece_id=N5H73PWM3J4TJ7IAF6Y2D5YBOE4N6TEDTOKDXNKISAHE5QU7UKUA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.630822831 (565h17m11.123255268s) [noise=true , address=45.91.116.12:4225 , piece_id=NDHKLBODOXW7I6BVSUYSI4HD7JIA7RIENQTHGACCHZVWJV7OUZAQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.634925323 (565h17m11.127357763s) [noise=true , address=45.91.116.8:4045 , piece_id=FMGLHYRQQAE3BFWFOL7H6Z52SHWC4VAFH6IHO7IFJBHRGR7KRBRQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.635470447 (565h17m11.12790288s) [noise=true , address=77.90.31.19:4841 , piece_id=XKJT2XATWPNLIWJYBZAOJAXAK2LQLTYBFF6QY3T67OYNVRDWPBMA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.637898075 (565h17m11.130330508s) [noise=true , address=45.91.116.13:4155 , piece_id=LDJUFT6P32JN6JG7X37OLMQIDB3YOGBVHINLI4PLMNZYF2U7OWVA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.640511390 (565h17m11.132943826s) [noise=true , address=77.90.31.10:4518 , piece_id=Q3ADWC5FIKIPHW62GZYUJSA2A5KEIBSTSA44QYGHIG7LKKU7WV3Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.642303091 (565h17m11.134735523s) [noise=true , address=45.91.116.7:4011 , piece_id=72BQFEAUQAK3N7B5NIRM4FUSGLSNO6KJW7GDBYT32U3FAI3OLZEQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.652686616 (565h17m11.145119058s) [noise=true , address=77.90.31.19:4843 , piece_id=WABF6QYOWYRBKAETKWVTFENKFXFHBZGDHO45NM4C4HAQNZTSNL3A, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.654812594 (565h17m11.14724503s) [noise=true , address=45.91.116.4:4940 , piece_id=4OETXZLBJNC4LOCW4JZSEETA6TCOKE6VBTKSQNZXFBGF5XCBMMXA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:50.668718247 (565h17m11.161150738s) [noise=true , address=77.90.31.15:4698 , piece_id=QCEUAGDNIOYBBXHD3GNPCWVG6MG5VFHKCEUHLXJJ3SDBWXRAGDJQ, seg_pos={0 0}, upload=497309]: Done uploading piece. err: <nil> Sep 22 03:24:52.419474418 (565h17m12.911906861s) [noise=true , address=145.79.227.16:4910 , piece_id=5OWHUIEMBGTSEELOPIVPIFIRC3NOT6TH2PMBXFHI5UR24VCGUQHA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:52.419494168 (565h17m12.911926604s) [noise=true , address=145.79.227.16:4909 , piece_id=OJXZAJ2KMOWV55PUE5YIL36HUY4N5ZO267PUW2JBZKWWBZ5RBDJA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448227337 (565h17m39.940659785s) [noise=true , address=145.79.227.5:4221 , piece_id=H5VJX26IHETQULTOMWQ5MH2RKRHDIPG75KVIGUNS7PRITJH6AB6Q, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 22 03:25:19.448346534 (565h17m39.940778963s) [noise=true , address=145.79.227.12:4767 , piece_id=W6KQ4VTOAWVWBBEWSLO7SLGZT7C5YHIPMSQIAFTWCRVK7E6XKHGA, seg_pos={0 0}, upload=497309]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:25:19.448373202 (565h17m39.940805632s) [seg_pos={0 0}, upload=497309]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:25:19.549520173 (565h17m40.041952616s) [upload=497309]: Done uploading segments... Sep 22 03:25:19.549557698 (565h17m40.041990148s) [upload=497309]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.103551102s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XX....X..........X.......X....X.X.......XX....X..X...X...] duration: 2.803504454s FAIL (2025-09-22 03:24:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.803531463s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X.X....X.....XX......X..................................X] duration: 30.112955328s FAIL (2025-09-22 03:24:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 03:23:54.634538079 (565h16m15.12697052s) [upload=497297]: Starting upload Sep 22 03:23:54.634646960 (565h16m15.127079408s) [upload=497297]: Done starting upload Sep 22 03:23:54.634797709 (565h16m15.127230145s) [upload=497297]: Uploading segments... Sep 22 03:23:54.635030991 (565h16m15.127463439s) [upload=497297]: Got next segment. Inline: false Sep 22 03:23:54.635080395 (565h16m15.127512835s) [upload=497297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 03:23:54.635088861 (565h16m15.127521296s) [upload=497297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 03:23:54.793600616 (565h16m15.286033049s) [seg_pos={0 0}, upload=497297]: Begin upload segment... Sep 22 03:23:54.793660471 (565h16m15.28609291s) [seg_pos={0 0}, upload=497297]: Done begin upload segment. Sep 22 03:23:54.793691575 (565h16m15.286124025s) [noise=true , address=45.91.116.10:4083 , piece_id=7JGBKJIM4HFOHRHPLINAKSU374FQX4QZS7CKVJIKF4EKAD325ESQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.794330430 (565h16m15.286762877s) [noise=true , address=45.91.116.4:4940 , piece_id=C6EQO2GMFU7FCOCWS2S7WAOG4MT7GYBKCNP2YZWZPYEOEKLUHHBQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.794716268 (565h16m15.287148711s) [noise=true , address=45.91.116.8:4047 , piece_id=ED5BA3SHKYSXEYSETWB44JTDFJGJ43HW2XISR6B2JTMLRH2P5N5Q, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.794988679 (565h16m15.287421124s) [noise=true , address=45.91.116.13:4153 , piece_id=PUNPNHAIB5BZRSIOPFPVRWF5GQCASX4B6BRGNORNXMI4NBX5YASQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.795266466 (565h16m15.287698907s) [noise=true , address=45.91.116.8:4048 , piece_id=NYPM7UASOK5VHVCN7LDTBABWBTZGTKHLGPEUEXWG25CGQVPC73AQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.795577564 (565h16m15.288010009s) [noise=true , address=45.91.116.5:4298 , piece_id=2TQMXODIX57YX6RLXZI2KXLRSVSRWUCO6XNELCJ2YZ6ZBEREWNXA, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.795902016 (565h16m15.288334465s) [noise=true , address=45.91.116.9:4262 , piece_id=NWW7Q3EXCUXV3NHNI5NRLIGBHYNQVZCMV2THRFF2K3ZQ5CTD5FGQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.796260181 (565h16m15.288692627s) [noise=true , address=77.90.31.17:4772 , piece_id=BCXG73MWV6HB32F6F5Z26FHY4IGSHPP6HUAU3S6I5DQ5E4AWA3CA, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.796593552 (565h16m15.289025999s) [noise=true , address=45.91.116.6:4974 , piece_id=4HXTXIUYN24HF6CQ2PH6HHHC5TXORF4BJ6VAMI77QGIE4TE2RWGQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.796875367 (565h16m15.289307812s) [noise=true , address=145.79.227.14:4839 , piece_id=GLCTVVUOYBJKO56SUXYFY7G7EKCDCBFMZUKP4RZORCMLSJWM7Q4Q, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.797209810 (565h16m15.289642258s) [noise=true , address=45.91.116.12:4226 , piece_id=QTQNEZDYPGZAPWD763TY3LUYZQBEB5Z2VXHF2II4ISXJOUCTT6FQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.797515839 (565h16m15.289948287s) [noise=true , address=77.90.31.19:4841 , piece_id=V4EPTO4BK75MFXJ65JXARKSSB7LWPWXPOP5T2GE6XYQQSY5G3LUA, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.797812577 (565h16m15.290245023s) [noise=true , address=145.79.227.15:4873 , piece_id=7FTNATLR4I66BY5BIYJVVM4Q6LAFXTELVFXJJ66LSOGJ2F3S6DLA, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.798176423 (565h16m15.290608877s) [noise=true , address=45.91.116.14:4189 , piece_id=6DOHE5HE5NLU7ZQ3CBJFOUDZO3FR7PSYM6RQFNOMQ6GIMBE26UKQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.798587658 (565h16m15.291020107s) [noise=true , address=145.79.227.19:4079 , piece_id=DBGSTAIYCCYHOGQRQHYEJJVFENYJDY7CASGO7EVRW22HBEWUPABQ, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:54.798967129 (565h16m15.291399576s) [noise=true , address=145.79.227.12:4767 , piece_id=3LCHMICQX5O4YMXAE726JFBLBH45JCXEHFFNMYS5UDPU6ZMHDWEA, seg_pos={0 0}, upload=497297]: Uploading piece... Sep 22 03:23:55.415895706 (565h16m15.908328149s) [upload=497297]: Next returned nil segment Sep 22 03:23:55.415913735 (565h16m15.908346187s) [upload=497297]: Waiting for error group managing segments... Sep 22 03:23:55.552094183 (565h16m16.044526618s) [noise=true , address=77.90.31.19:4841 , piece_id=V4EPTO4BK75MFXJ65JXARKSSB7LWPWXPOP5T2GE6XYQQSY5G3LUA, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.552583787 (565h16m16.045016219s) [noise=true , address=45.91.116.12:4226 , piece_id=QTQNEZDYPGZAPWD763TY3LUYZQBEB5Z2VXHF2II4ISXJOUCTT6FQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.553130032 (565h16m16.045562465s) [noise=true , address=45.91.116.9:4262 , piece_id=NWW7Q3EXCUXV3NHNI5NRLIGBHYNQVZCMV2THRFF2K3ZQ5CTD5FGQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.556608527 (565h16m16.049040962s) [noise=true , address=45.91.116.8:4048 , piece_id=NYPM7UASOK5VHVCN7LDTBABWBTZGTKHLGPEUEXWG25CGQVPC73AQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.557180241 (565h16m16.049612676s) [noise=true , address=45.91.116.13:4153 , piece_id=PUNPNHAIB5BZRSIOPFPVRWF5GQCASX4B6BRGNORNXMI4NBX5YASQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.557592913 (565h16m16.050025349s) [noise=true , address=45.91.116.4:4940 , piece_id=C6EQO2GMFU7FCOCWS2S7WAOG4MT7GYBKCNP2YZWZPYEOEKLUHHBQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.558988019 (565h16m16.051420451s) [noise=true , address=45.91.116.6:4974 , piece_id=4HXTXIUYN24HF6CQ2PH6HHHC5TXORF4BJ6VAMI77QGIE4TE2RWGQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.561899913 (565h16m16.054332345s) [noise=true , address=77.90.31.17:4772 , piece_id=BCXG73MWV6HB32F6F5Z26FHY4IGSHPP6HUAU3S6I5DQ5E4AWA3CA, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.563931232 (565h16m16.05636366s) [noise=true , address=45.91.116.14:4189 , piece_id=6DOHE5HE5NLU7ZQ3CBJFOUDZO3FR7PSYM6RQFNOMQ6GIMBE26UKQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.565056163 (565h16m16.057488596s) [noise=true , address=45.91.116.10:4083 , piece_id=7JGBKJIM4HFOHRHPLINAKSU374FQX4QZS7CKVJIKF4EKAD325ESQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.566833367 (565h16m16.0592658s) [noise=true , address=45.91.116.5:4298 , piece_id=2TQMXODIX57YX6RLXZI2KXLRSVSRWUCO6XNELCJ2YZ6ZBEREWNXA, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:23:55.567895649 (565h16m16.060328078s) [noise=true , address=45.91.116.8:4047 , piece_id=ED5BA3SHKYSXEYSETWB44JTDFJGJ43HW2XISR6B2JTMLRH2P5N5Q, seg_pos={0 0}, upload=497297]: Done uploading piece. err: <nil> Sep 22 03:24:24.334772687 (565h16m44.827205131s) [noise=true , address=145.79.227.12:4767 , piece_id=3LCHMICQX5O4YMXAE726JFBLBH45JCXEHFFNMYS5UDPU6ZMHDWEA, seg_pos={0 0}, upload=497297]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:24.334848206 (565h16m44.82728064s) [noise=true , address=145.79.227.14:4839 , piece_id=GLCTVVUOYBJKO56SUXYFY7G7EKCDCBFMZUKP4RZORCMLSJWM7Q4Q, seg_pos={0 0}, upload=497297]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:24.334881263 (565h16m44.827313698s) [noise=true , address=145.79.227.19:4079 , piece_id=DBGSTAIYCCYHOGQRQHYEJJVFENYJDY7CASGO7EVRW22HBEWUPABQ, seg_pos={0 0}, upload=497297]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: rpc: dial tcp 145.79.227.19:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:24.334904370 (565h16m44.827336801s) [noise=true , address=145.79.227.15:4873 , piece_id=7FTNATLR4I66BY5BIYJVVM4Q6LAFXTELVFXJJ66LSOGJ2F3S6DLA, seg_pos={0 0}, upload=497297]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 03:24:24.334948739 (565h16m44.827381172s) [seg_pos={0 0}, upload=497297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 03:24:24.444985307 (565h16m44.93741774s) [upload=497297]: Done uploading segments... Sep 22 03:24:24.445045439 (565h16m44.937477892s) [upload=497297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.112995199s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X...XX....X..........X.......X....X.X.......XX....X..X...X..] duration: 2.771757051s FAIL (2025-09-22 03:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.771788048s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..X.X....X.....XX......X..................................] duration: 2.366009339s FAIL (2025-09-22 03:23:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.366037988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..X.X....X.....XX......X.................................] duration: 2.465261663s FAIL (2025-09-22 03:23:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.465295636s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X...XX....X..........X.......X....X.X.......XX....X..X...X.] duration: 2.820781778s FAIL (2025-09-22 03:23:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.820801999s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X...XX....X..........X.......X....X.X.......XX....X..X...X] duration: 30.000647412s FAIL (2025-09-22 03:22:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000690177s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X..X.X....X.....XX......X................................] duration: 2.515706126s FAIL (2025-09-22 03:22:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.515820262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..X..X.X....X.....XX......X...............................] duration: 2.65284381s FAIL (2025-09-22 03:21:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.65292444s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X...XX....X..........X.......X....X.X.......XX....X..X...] duration: 2.800350773s FAIL (2025-09-22 03:21:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.800378973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..X..X.X....X.....XX......X..............................] duration: 2.571353916s FAIL (2025-09-22 03:21:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.571389659s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X...XX....X..........X.......X....X.X.......XX....X..X..] duration: 2.75453357s FAIL (2025-09-22 03:20:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.754793206s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..X..X.X....X.....XX......X.............................] duration: 2.604201713s FAIL (2025-09-22 03:20:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.604226377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..X..X.X....X.....XX......X............................] duration: 2.366522488s FAIL (2025-09-22 03:19:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.366552355s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X...XX....X..........X.......X....X.X.......XX....X..X.] duration: 2.753967051s FAIL (2025-09-22 03:19:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.753996546s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.X...XX....X..........X.......X....X.X.......XX....X..X] duration: 30.001380296s FAIL (2025-09-22 03:19:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001407278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X..X..X.X....X.....XX......X...........................] duration: 2.423021366s FAIL (2025-09-22 03:18:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.423059397s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X...XX....X..........X.......X....X.X.......XX....X..] duration: 2.759867704s FAIL (2025-09-22 03:18:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.759885884s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X..X..X.X....X.....XX......X..........................] duration: 2.373377389s FAIL (2025-09-22 03:17:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373407383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X..X..X.X....X.....XX......X.........................] duration: 2.523116912s FAIL (2025-09-22 03:16:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.52315418s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.....X.X...XX....X..........X.......X....X.X.......XX....X.] duration: 2.996420242s FAIL (2025-09-22 03:16:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.996468015s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.....X.X...XX....X..........X.......X....X.X.......XX....X] duration: 30.001408183s FAIL (2025-09-22 03:16:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001433151s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X..X..X.X....X.....XX......X........................] duration: 2.355975844s FAIL (2025-09-22 03:15:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.356005249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X..X..X.X....X.....XX......X.......................] duration: 2.56300928s FAIL (2025-09-22 03:15:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.563034925s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.....X.X...XX....X..........X.......X....X.X.......XX....] duration: 3.053728494s FAIL (2025-09-22 03:15:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.053997134s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X.....X.X...XX....X..........X.......X....X.X.......XX...] duration: 2.666068498s FAIL (2025-09-22 03:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.666096781s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.........X..X..X.X....X.....XX......X......................] duration: 2.433269325s FAIL (2025-09-22 03:14:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.433300832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX.........X..X..X.X....X.....XX......X.....................] duration: 2.49598528s FAIL (2025-09-22 03:14:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.496071148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX.........X..X..X.X....X.....XX......X....................] duration: 2.380420759s FAIL (2025-09-22 03:13:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.380454052s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X.....X.X...XX....X..........X.......X....X.X.......XX..] duration: 2.744959681s FAIL (2025-09-22 03:13:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.745003931s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X.....X.X...XX....X..........X.......X....X.X.......XX.] duration: 2.881660594s FAIL (2025-09-22 03:12:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.881674644s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.X.....X.X...XX....X..........X.......X....X.X.......XX] duration: 30.001376493s FAIL (2025-09-22 03:12:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001400308s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.XX.........X..X..X.X....X.....XX......X...................] duration: 2.424657353s FAIL (2025-09-22 03:12:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.424687876s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X.....X.X...XX....X..........X.......X....X.X.......X] duration: 30.002446517s FAIL (2025-09-22 03:11:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002502819s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.XX.........X..X..X.X....X.....XX......X..................] duration: 2.425039384s FAIL (2025-09-22 03:11:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.425080812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X.XX.........X..X..X.X....X.....XX......X.................] duration: 2.410349389s FAIL (2025-09-22 03:11:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:58:53.173370076 (564h51m13.665802511s) [upload=496929]: Starting upload Sep 22 02:58:53.173441332 (564h51m13.665873766s) [upload=496929]: Done starting upload Sep 22 02:58:53.173615843 (564h51m13.666048279s) [upload=496929]: Uploading segments... Sep 22 02:58:53.173840170 (564h51m13.666272605s) [upload=496929]: Got next segment. Inline: false Sep 22 02:58:53.173974513 (564h51m13.666406945s) [upload=496929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:58:53.173981617 (564h51m13.66641405s) [upload=496929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:58:53.303293151 (564h51m13.795725585s) [seg_pos={0 0}, upload=496929]: Begin upload segment... Sep 22 02:58:53.303361348 (564h51m13.795793782s) [seg_pos={0 0}, upload=496929]: Done begin upload segment. Sep 22 02:58:53.303414885 (564h51m13.795847336s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.303558271 (564h51m13.795990755s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304004988 (564h51m13.796437443s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304049159 (564h51m13.796481602s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304389164 (564h51m13.796821611s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.304753712 (564h51m13.797186166s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305125310 (564h51m13.797557759s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305149026 (564h51m13.797581483s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305522594 (564h51m13.797955063s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305591142 (564h51m13.798023589s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.305898259 (564h51m13.798330709s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.306081282 (564h51m13.798513766s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307266454 (564h51m13.799698909s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.307733666 (564h51m13.8001661s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308122207 (564h51m13.800554674s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.308579493 (564h51m13.801011947s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Uploading piece... Sep 22 02:58:53.931834493 (564h51m14.424266927s) [upload=496929]: Next returned nil segment Sep 22 02:58:53.931855881 (564h51m14.424288331s) [upload=496929]: Waiting for error group managing segments... Sep 22 02:58:54.063098864 (564h51m14.555531319s) [noise=true , address=45.91.116.4:4937 , piece_id=4J75ND7ALO6IVO7LZ3Z2ZYYFHMUM36GTBNSTYONFHOKHTR5PABGQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.064241799 (564h51m14.556674254s) [noise=true , address=45.91.116.11:4118 , piece_id=WU7UO5PP2O3F6UBECQMDGVPNG34KMX2XHEEV6YWZ7B2HCITS623A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065313453 (564h51m14.557745897s) [noise=true , address=45.91.116.8:4048 , piece_id=LQBNML62VBCEMCD7PRCGZUBXYEJKPPQIXHXHXXVQCNV5ZRYH37TA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.065950698 (564h51m14.558383146s) [noise=true , address=77.90.31.15:4699 , piece_id=GRN4C4NVOMY7RPWXHIW3IZLHYEQKWXBVUDX22SUCQWHPTJGAT2YA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.066200099 (564h51m14.558632533s) [noise=true , address=45.91.116.9:4263 , piece_id=UU5SAA7UZBTVGEYMLI5A2GL3FPOGJ24K4IWCPJN23YBYK27N6XTQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.067040587 (564h51m14.559473039s) [noise=true , address=45.91.116.14:4190 , piece_id=VI5LFRTREHVXTL4AY3BKMFCA4WUDGZ7H4W6SHZXF64J557456T7A, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.068376638 (564h51m14.560809068s) [noise=true , address=77.90.31.12:4589 , piece_id=KRIQJV2LONSA7EYIA27CBZ3FU5JEYJML5N6E7A5DGIIT75RCI74Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070220883 (564h51m14.562653324s) [noise=true , address=45.91.116.6:4974 , piece_id=SB2VSMXBKUGOJ45WYC45HSHYYCJTOTPQ5C5BMTYM2FS5P4MT3PHQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.070537309 (564h51m14.562969759s) [noise=true , address=77.90.31.16:4735 , piece_id=5D65HDRXN3S42HYDBRVZZD2R54OMDF6IIKV5NC7DN5GH3GRXV7NA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.074625189 (564h51m14.567057631s) [noise=true , address=77.90.31.21:4878 , piece_id=4GHR2RONDN2GI3Y5HWCN5AQKMFMX7GTGMRZUTOQIKK2HVK6M2EFA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080466955 (564h51m14.572899384s) [noise=true , address=77.90.31.13:4627 , piece_id=RMOZIX3NJRACRQUYDJXEHRORMXKVMPWNRU7GDYFQOZRX5RK5BOIA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:58:54.080529362 (564h51m14.572961796s) [noise=true , address=77.90.31.10:4520 , piece_id=FEBZLXPERUILXU3T6JHDVZAPYEB7J6YVLD6VUWFAUKRVOHQ33XLQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: <nil> Sep 22 02:59:22.874452783 (564h51m43.366885226s) [noise=true , address=145.79.227.15:4874 , piece_id=CGAYBT5W6OEMYEK3RH3VIPKPJ63D4BQVWIEFZOPA4YC4MFZXLVMQ, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: rpc: dial tcp 145.79.227.15:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874482569 (564h51m43.366915011s) [noise=true , address=145.79.227.19:4077 , piece_id=3HOSLKQTFURJMRY6KSYMCERWXA5ULWHMIRPW34QTGCXAPYNPKISA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874530941 (564h51m43.366963383s) [noise=true , address=145.79.227.14:4839 , piece_id=3BL7JFMSJNY3LMISAOHAMJ7JNS6SQ7BMORWR4GNGBOU5AHK5XBGA, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874539810 (564h51m43.366972242s) [noise=true , address=145.79.227.13:4801 , piece_id=V5EYSAAAUQVWG2PA4SHWDGLARQYNUZU62YRCAIA7YUF72TW5MB4Q, seg_pos={0 0}, upload=496929]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:59:22.874585846 (564h51m43.36701829s) [seg_pos={0 0}, upload=496929]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:59:23.007289986 (564h51m43.499722422s) [upload=496929]: Done uploading segments... Sep 22 02:59:23.007319383 (564h51m43.499751825s) [upload=496929]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.410389513s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)