Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
40 out of 60 is failed (threshold=3) [X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X.] duration:
2.916312298s
FAIL (2025-07-09 02:59:11 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.916654797s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
40 out of 60 is failed (threshold=3) [.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X] duration:
1m0.000883023s
FAIL (2025-07-09 02:59:08 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000911272s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X...................................X............] duration:
2.483759764s
FAIL (2025-07-09 02:58:57 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.48381614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [......XX..X........X.....XXX.X.XXX....X..X.X.X.X............] duration:
2.63866469s
FAIL (2025-07-09 02:58:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.638693413s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X...................................X...........] duration:
2.34737963s
FAIL (2025-07-09 02:58:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.347423913s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.......XX..X........X.....XXX.X.XXX....X..X.X.X.X...........] duration:
2.820699441s
FAIL (2025-07-09 02:58:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.820760697s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X...........................X.X............................] duration:
2.289148991s
FAIL (2025-07-09 02:58:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.289177067s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [........XX..X........X.....XXX.X.XXX....X..X.X.X.X..........] duration:
3.583793735s
FAIL (2025-07-09 02:57:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.583824041s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X...........................X.X...........................] duration:
2.34794398s
FAIL (2025-07-09 02:57:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.347979648s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............X...................................X..........] duration:
11.683788273s
FAIL (2025-07-09 02:57:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 11.683878645s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
40 out of 60 is failed (threshold=3) [X.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...] duration:
6.756699441s
FAIL (2025-07-09 02:57:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.756756228s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X...........................X.X..........................] duration:
12.605338943s
FAIL (2025-07-09 02:57:04 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 12.605371037s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [XX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX..] duration:
44.611739697s
FAIL (2025-07-09 02:57:02 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 44.611758059s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.........XX..X........X.....XXX.X.XXX....X..X.X.X.X.........] duration:
2.556408591s
FAIL (2025-07-09 02:56:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.556452808s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..........XX..X........X.....XXX.X.XXX....X..X.X.X.X........] duration:
20.382675653s
FAIL (2025-07-09 02:56:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.382703533s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............X...................................X.........] duration:
15.240053547s
FAIL (2025-07-09 02:56:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 15.24014427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X...........................X.X.........................] duration:
2.329636972s
FAIL (2025-07-09 02:56:25 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.329679863s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX.] duration:
44.842830945s
FAIL (2025-07-09 02:56:17 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 44.842844439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............X...................................X........] duration:
16.944351002s
FAIL (2025-07-09 02:56:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 16.944385372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...........XX..X........X.....XXX.X.XXX....X..X.X.X.X.......] duration:
6.184013597s
FAIL (2025-07-09 02:55:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.184028628s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX] duration:
1m0.002305687s
FAIL (2025-07-09 02:55:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.002333837s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X...........................X.X........................] duration:
2.280882025s
FAIL (2025-07-09 02:55:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.280911509s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [............XX..X........X.....XXX.X.XXX....X..X.X.X.X......] duration:
23.960334518s
FAIL (2025-07-09 02:55:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.960358905s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X...........................X.X.......................] duration:
2.446656205s
FAIL (2025-07-09 02:55:06 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.446710686s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.............XX..X........X.....XXX.X.XXX....X..X.X.X.X.....] duration:
7.375574847s
FAIL (2025-07-09 02:54:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.375596599s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [X...............X...................................X.......] duration:
3.480683207s
FAIL (2025-07-09 02:54:42 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.480712699s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [X.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXX] duration:
1m0.000930267s
FAIL (2025-07-09 02:54:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000954544s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.X...............X...................................X......] duration:
49.694981363s
FAIL (2025-07-09 02:54:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 49.695018033s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X...........................X.X......................] duration:
9.821769382s
FAIL (2025-07-09 02:54:13 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 9.821800201s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..............XX..X........X.....XXX.X.XXX....X..X.X.X.X....] duration:
2.73251603s
FAIL (2025-07-09 02:53:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.732538694s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X...........................X.X.....................] duration:
2.983635434s
FAIL (2025-07-09 02:53:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.983664485s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [XX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XX] duration:
1m0.001244505s
FAIL (2025-07-09 02:53:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001267529s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X........X...........................X.X....................] duration:
2.724371335s
FAIL (2025-07-09 02:53:28 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.7243997s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..X...............X...................................X.....] duration:
36.165154532s
FAIL (2025-07-09 02:53:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 36.165185163s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...............XX..X........X.....XXX.X.XXX....X..X.X.X.X...] duration:
3.490707858s
FAIL (2025-07-09 02:52:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.490734894s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X........X...........................X.X...................] duration:
2.285170514s
FAIL (2025-07-09 02:52:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.285209761s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.X] duration:
1m0.001075696s
FAIL (2025-07-09 02:52:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001092134s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X........X...........................X.X..................] duration:
2.581105787s
FAIL (2025-07-09 02:51:55 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.581138488s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [................XX..X........X.....XXX.X.XXX....X..X.X.X.X..] duration:
21.737873156s
FAIL (2025-07-09 02:51:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.737899122s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [.XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.] duration:
52.460005298s
FAIL (2025-07-09 02:51:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 52.460029092s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X...............X...................................X....] duration:
3.394444557s
FAIL (2025-07-09 02:51:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.394477054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.................XX..X........X.....XXX.X.XXX....X..X.X.X.X.] duration:
22.165592461s
FAIL (2025-07-09 02:51:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.165626408s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X........X...........................X.X.................] duration:
2.383874819s
FAIL (2025-07-09 02:51:05 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.383907677s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X........X...........................X.X................] duration:
17.614933062s
FAIL (2025-07-09 02:51:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 17.614966966s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X...............X...................................X...] duration:
17.304820274s
FAIL (2025-07-09 02:50:48 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.304851003s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.................XX..X........X.....XXX.X.XXX....X..X.X.X.X] duration:
30.000566988s
FAIL (2025-07-09 02:50:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00059227s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X...............X...................................X..] duration:
17.236391451s
FAIL (2025-07-09 02:50:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.236419035s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X] duration:
1m0.001096827s
FAIL (2025-07-09 02:50:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001130427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X...............X...................................X.] duration:
51.639715255s
FAIL (2025-07-09 02:50:05 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 51.63975714s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X........X...........................X.X...............] duration:
12.771033689s
FAIL (2025-07-09 02:49:46 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 12.771068558s
core
uplink
us-select-1
|