Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......X...............X...................................X] duration:
1m0.110780594s
FAIL (2025-07-09 02:49:13 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.110796757s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X.................XX..X........X.....XXX.X.XXX....X..X.X.X.] duration:
5.393859478s
FAIL (2025-07-09 02:49:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.393893665s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
41 out of 60 is failed (threshold=3) [X..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.] duration:
52.909632696s
FAIL (2025-07-09 02:49:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 52.909651397s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X........X...........................X.X..............] duration:
2.338012838s
FAIL (2025-07-09 02:48:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.338040549s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..X.................XX..X........X.....XXX.X.XXX....X..X.X.X] duration:
30.001421015s
FAIL (2025-07-09 02:48:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001450389s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X........X...........................X.X.............] duration:
2.819077252s
FAIL (2025-07-09 02:48:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.819109015s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........X...............X...................................] duration:
37.610998319s
FAIL (2025-07-09 02:48:13 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 37.611048338s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [XX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX] duration:
1m0.000688486s
FAIL (2025-07-09 02:48:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000701836s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...X.................XX..X........X.....XXX.X.XXX....X..X.X.] duration:
2.998818442s
FAIL (2025-07-09 02:47:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.998849622s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X........X...........................X.X............] duration:
3.641909621s
FAIL (2025-07-09 02:47:25 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 3.641940494s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X...............X..................................] duration:
48.47229044s
FAIL (2025-07-09 02:47:11 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 48.472386551s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [XXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.X] duration:
1m0.001225612s
FAIL (2025-07-09 02:47:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001255554s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [....X.................XX..X........X.....XXX.X.XXX....X..X.X] duration:
30.001176184s
FAIL (2025-07-09 02:47:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001202071s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X........X........X...........................X.X...........] duration:
16.887231733s
FAIL (2025-07-09 02:46:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 16.887278556s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.....X.................XX..X........X.....XXX.X.XXX....X..X.] duration:
15.164019844s
FAIL (2025-07-09 02:46:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.164055322s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........X...............X.................................] duration:
38.683027517s
FAIL (2025-07-09 02:46:08 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 38.683064279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.] duration:
46.776277796s
FAIL (2025-07-09 02:46:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 46.776295983s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X...............X................................] duration:
36.450547541s
FAIL (2025-07-09 02:45:21 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 36.450571693s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [......X.................XX..X........X.....XXX.X.XXX....X..X] duration:
30.001102453s
FAIL (2025-07-09 02:45:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001146554s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X] duration:
1m0.001471428s
FAIL (2025-07-09 02:45:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00149766s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X........X........X...........................X.X..........] duration:
2.355402483s
FAIL (2025-07-09 02:45:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.355432791s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X...............X...............................] duration:
37.475866726s
FAIL (2025-07-09 02:44:45 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 37.475890703s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.......X.................XX..X........X.....XXX.X.XXX....X..] duration:
5.658148709s
FAIL (2025-07-09 02:44:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.658199444s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X........X........X...........................X.X.........] duration:
2.378913531s
FAIL (2025-07-09 02:44:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.378941115s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............X...............X..............................] duration:
33.500553152s
FAIL (2025-07-09 02:44:07 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 33.500573264s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [........X.................XX..X........X.....XXX.X.XXX....X.] duration:
23.570541677s
FAIL (2025-07-09 02:43:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.570571622s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
42 out of 60 is failed (threshold=3) [X.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..] duration:
36.85538937s
FAIL (2025-07-09 02:43:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 36.855408767s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............X...............X.............................] duration:
33.485733621s
FAIL (2025-07-09 02:43:34 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 33.485759099s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X........X........X...........................X.X........] duration:
2.292709692s
FAIL (2025-07-09 02:43:10 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.292739377s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX.] duration:
56.776889785s
FAIL (2025-07-09 02:43:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 56.77690313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.........X.................XX..X........X.....XXX.X.XXX....X] duration:
30.001730225s
FAIL (2025-07-09 02:43:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001761687s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............X...............X............................] duration:
36.958209137s
FAIL (2025-07-09 02:43:00 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 36.95823682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X........X........X...........................X.X.......] duration:
2.435424604s
FAIL (2025-07-09 02:42:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.435484638s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX] duration:
1m0.000848883s
FAIL (2025-07-09 02:42:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000863656s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................X...............X...........................] duration:
25.571346088s
FAIL (2025-07-09 02:41:48 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 25.571378505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X........X........X...........................X.X......] duration:
10.999158372s
FAIL (2025-07-09 02:41:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 10.999196932s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X.........X.................XX..X........X.....XXX.X.XXX....] duration:
2.684961889s
FAIL (2025-07-09 02:41:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.684984709s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X........X........X...........................X.X.....] duration:
24.28742054s
FAIL (2025-07-09 02:41:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 24.287463349s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X.........X.................XX..X........X.....XXX.X.XXX...] duration:
22.70689956s
FAIL (2025-07-09 02:41:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.70691282s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [X.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XX] duration:
1m0.00104025s
FAIL (2025-07-09 02:41:09 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001057687s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................X...............X..........................] duration:
26.492924905s
FAIL (2025-07-09 02:40:46 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 26.492952846s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X........X........X...........................X.X....] duration:
2.318188669s
FAIL (2025-07-09 02:40:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.318228944s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................X...............X.........................] duration:
32.866735109s
FAIL (2025-07-09 02:40:19 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 32.866771681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X.........X.................XX..X........X.....XXX.X.XXX..] duration:
21.046943918s
FAIL (2025-07-09 02:40:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.046965114s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
43 out of 60 is failed (threshold=3) [XX.XX.XXXX..XXX.XXX.X.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.X] duration:
1m0.00110819s
FAIL (2025-07-09 02:40:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001185492s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X........X........X...........................X.X...] duration:
24.668190244s
FAIL (2025-07-09 02:40:05 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 24.668220376s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................X...............X........................] duration:
33.668423326s
FAIL (2025-07-09 02:39:31 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
Jul 9 02:18:54.004351146 (377h7m53.887764765s) [upload=301584]: Got next segment. Inline: false
Jul 9 02:18:54.004413603 (377h7m53.887827234s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:18:54.004419760 (377h7m53.887833387s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:18:54.213249552 (377h7m54.096663181s) [seg_pos={0 0}, upload=301584]: Begin upload segment...
Jul 9 02:18:54.213299816 (377h7m54.096713429s) [seg_pos={0 0}, upload=301584]: Done begin upload segment.
Jul 9 02:18:54.213329638 (377h7m54.096743275s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213340182 (377h7m54.096753808s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213411091 (377h7m54.096824717s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213737606 (377h7m54.097151235s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.213747431 (377h7m54.097161064s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214106156 (377h7m54.097519795s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214221185 (377h7m54.097634815s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214418807 (377h7m54.097832427s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214605987 (377h7m54.098019613s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.214984578 (377h7m54.098398214s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217026373 (377h7m54.100440074s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217407228 (377h7m54.100820857s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.217695328 (377h7m54.101108954s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218235663 (377h7m54.101649297s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218567500 (377h7m54.101981128s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.218847734 (377h7m54.10226137s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219157211 (377h7m54.102570837s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.219433241 (377h7m54.102846867s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Uploading piece...
Jul 9 02:18:54.779916970 (377h7m54.663330582s) [upload=301584]: Next returned nil segment
Jul 9 02:18:54.780041765 (377h7m54.663455401s) [upload=301584]: Waiting for error group managing segments...
Jul 9 02:18:54.906825760 (377h7m54.790239387s) [noise=true , address=45.91.116.4:4964 , piece_id=BK7L2MWHHHTHGXBJGETKNYA6TCPQPIIGRCLE6XM7ZLGXUQD5N5UQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: ecclient: upload failed (node:1gskLwXRsnfLFfLENJ8zY81DJwLCSaBzPPdi5BryyQ5iAwiVi7, address:45.91.116.4:4964): protocol: storage node overloaded, request limit: 2000
Jul 9 02:18:54.910040381 (377h7m54.793453998s) [noise=true , address=77.90.31.19:4860 , piece_id=COUDAWAMBEAMJ7NN2EXZXRJB22GNHJTVZPDRLHD2EKLRMO22IBVA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.911684313 (377h7m54.795097933s) [noise=true , address=77.90.31.18:4830 , piece_id=J2H3ILISDW3EM42RH46NJNK7MJIPD3KVPI5WSUL5L2STZVGWRELQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919143936 (377h7m54.802557549s) [noise=true , address=77.90.31.11:4577 , piece_id=HM7KLGU4ZBRSHCGXJKZQD3R3BMQWKIWXOKJZNQRLBNRPX5MBFBPA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.919247663 (377h7m54.80266128s) [noise=true , address=77.90.31.19:4858 , piece_id=Y3J25OVRVDSLS2Y4C4B64DU5YU7GSJSPGWHENASFEQD4JUYSTFLQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.922295100 (377h7m54.805708721s) [noise=true , address=77.90.31.16:4765 , piece_id=LRPUA7BUB5Z4HYHPRFO4ZGLDOYZJYJZ7O2LBXOXDDKXXXXLOMNWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.932112542 (377h7m54.815526158s) [noise=true , address=77.90.31.21:4905 , piece_id=AHLRZZDFAYG4O5UHPPCDUXUHKVWZAIKLNER4T6PKTPWRBVR66JNQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:54.985158729 (377h7m54.868572349s) [noise=true , address=79.127.219.48:4076 , piece_id=4F5XS4K6QVBXVL6UFYWYR4FQGFBSWWLGNQOOVUUJLRVPVUYCRA5A, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.252912801 (377h7m55.13632643s) [noise=true , address=156.146.43.225:4468 , piece_id=AVXV7TJL2Z7JJXEABTKER44ZN573EROPAONBWSE7L3OKK4URY27Q, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:18:55.794737257 (377h7m55.678150888s) [noise=true , address=45.91.116.7:4041 , piece_id=7CUFAUN2FCYAWYEB3KUDKORK7KLGFGGQCNPDJOZQWME36Y24JKEQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:04.769066043 (377h8m4.652479664s) [noise=true , address=104.36.82.10:4094 , piece_id=NFSXASST5YBICFJPMPF6C3W7VYIBVA7JPCS4Z7ONLUYEJLJWH6WQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:08.304400869 (377h8m8.18781454s) [noise=true , address=64.49.13.137:4497 , piece_id=RQKQBGBTMNEMU4TSWVHWJHSQF7VNX62HGUDLF3STDGEXV2IZLLWA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:30.319972582 (377h8m30.203386199s) [noise=true , address=104.36.82.5:4869 , piece_id=OFRRQO6XP3QWLGTFDEJGC4LMXUAGQZVT2PUPDW6I2R6EJTQ6STYQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:31.789221285 (377h8m31.672634897s) [noise=true , address=104.36.82.18:4369 , piece_id=27RHRAOXUITPUJIUPDWQFEY6E5YGKVYYA7BE4YFKIT3MLMVKBABA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302125144 (377h8m51.185538757s) [noise=true , address=104.36.82.21:4497 , piece_id=ZYM7PK6ZXPBABWCEGOCWVIHMGZD5AQ3ME5GZZMNSO4TBOGBBLZZA, seg_pos={0 0}, upload=301584]: Done uploading piece. err: <nil>
Jul 9 02:19:51.302145626 (377h8m51.185559254s) [seg_pos={0 0}, upload=301584]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:19:51.302256469 (377h8m51.185670081s) [noise=true , address=64.49.13.169:4706 , piece_id=2NPXS3FF2PTYQB7L6B5X2NU4QSJPJWMMBA4MVI26WWZLGUQFHCGQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302388338 (377h8m51.185801965s) [noise=true , address=104.36.82.60:4353 , piece_id=WMFKPZUZ5M3SOH2GU23ZVXYV5ELL3F6DOMYWW4AA2PHB5OUZSWTQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jul 9 02:19:51.302454505 (377h8m51.185868117s) [noise=true , address=104.36.82.39:4726 , piece_id=B2U65MTV25IOTJT3HXBBLRUDOLEPN4CAJTSI6QJ3JSN7WOOXZRDQ, seg_pos={0 0}, upload=301584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48412->104.36.82.39:4726: use of closed network connection
Jul 9 02:19:51.302478392 (377h8m51.185892005s) [seg_pos={0 0}, upload=301584]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:19:51.302544260 (377h8m51.185957887s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:19:51.302548542 (377h8m51.185962171s) [upload=301584]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:19:51.449672035 (377h8m51.333085675s) [upload=301584]: Done uploading segments...
Jul 9 02:19:51.449694100 (377h8m51.333107731s) [upload=301584]: Upload finished. err: <nil>
Duration: 33.66847401s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X........X........X...........................X.X..] duration:
2.428490635s
FAIL (2025-07-09 02:39:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.428524724s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...X.........X.................XX..X........X.....XXX.X.XXX.] duration:
3.121173729s
FAIL (2025-07-09 02:39:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.121312401s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [....X.........X.................XX..X........X.....XXX.X.XXX] duration:
30.001206024s
FAIL (2025-07-09 02:38:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001233927s
edge
gateway
us-select-1
|