Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
28 out of 60 is failed (threshold=3) [XXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X...............] duration:
3.423434708s
FAIL (2025-07-09 03:09:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.423468917s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration:
2.26243109s
FAIL (2025-07-09 03:08:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.262459905s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.494713553s
FAIL (2025-07-09 03:08:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.49474327s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.335483912s
FAIL (2025-07-09 03:08:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.335514977s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
29 out of 60 is failed (threshold=3) [XXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X..............] duration:
3.221120057s
FAIL (2025-07-09 03:08:24 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.221215861s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X.X.........................................] duration:
2.29563664s
FAIL (2025-07-09 03:08:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.29572785s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.......X.....XXX.X.XXX....X..X.X.X.X........................] duration:
2.762522601s
FAIL (2025-07-09 03:08:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.762566046s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
30 out of 60 is failed (threshold=3) [XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X.............] duration:
2.993762677s
FAIL (2025-07-09 03:08:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.994002768s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X...................................X.......................] duration:
2.385587073s
FAIL (2025-07-09 03:07:57 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.385616325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [........X.....XXX.X.XXX....X..X.X.X.X.......................] duration:
2.656376442s
FAIL (2025-07-09 03:07:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.656504169s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
30 out of 60 is failed (threshold=3) [.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X............] duration:
3.374826388s
FAIL (2025-07-09 03:07:43 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.374850712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X........X.....XXX.X.XXX....X..X.X.X.X......................] duration:
2.811359492s
FAIL (2025-07-09 03:07:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.811385425s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X...................................X......................] duration:
2.387405131s
FAIL (2025-07-09 03:07:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.387444408s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X.X........................................] duration:
2.28513934s
FAIL (2025-07-09 03:07:11 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.285171929s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X........X.....XXX.X.XXX....X..X.X.X.X.....................] duration:
3.456589654s
FAIL (2025-07-09 03:07:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.456705785s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
31 out of 60 is failed (threshold=3) [X.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X...........] duration:
2.973570623s
FAIL (2025-07-09 03:06:47 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.973593941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X...................................X.....................] duration:
2.482683911s
FAIL (2025-07-09 03:06:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.482712792s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration:
3.41773165s
FAIL (2025-07-09 03:06:23 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 3.41776699s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X........X.....XXX.X.XXX....X..X.X.X.X....................] duration:
2.559823917s
FAIL (2025-07-09 03:06:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.559930314s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
32 out of 60 is failed (threshold=3) [XX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X..........] duration:
2.927975231s
FAIL (2025-07-09 03:05:51 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.927999611s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X...................................X....................] duration:
2.498011307s
FAIL (2025-07-09 03:05:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.498063735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X.X......................................] duration:
2.339679149s
FAIL (2025-07-09 03:05:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.339714519s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X...................................X...................] duration:
7.848949136s
FAIL (2025-07-09 03:05:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 7.84898828s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
33 out of 60 is failed (threshold=3) [XXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X.........] duration:
3.320806118s
FAIL (2025-07-09 03:05:06 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.320923139s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration:
2.276680543s
FAIL (2025-07-09 03:04:58 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.276717445s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X..X........X.....XXX.X.XXX....X..X.X.X.X...................] duration:
2.614655038s
FAIL (2025-07-09 03:04:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.614675989s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration:
2.316375989s
FAIL (2025-07-09 03:04:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.316485912s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....X...................................X..................] duration:
2.576502189s
FAIL (2025-07-09 03:04:23 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.576542926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
34 out of 60 is failed (threshold=3) [XXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X........] duration:
6.781980485s
FAIL (2025-07-09 03:04:18 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.782008531s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XX..X........X.....XXX.X.XXX....X..X.X.X.X..................] duration:
2.580754258s
FAIL (2025-07-09 03:03:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.580790831s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......X...................................X.................] duration:
7.599103518s
FAIL (2025-07-09 03:03:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 7.599128084s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
35 out of 60 is failed (threshold=3) [XXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X.......] duration:
4.579125532s
FAIL (2025-07-09 03:03:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.579313631s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X.X...................................] duration:
2.303010317s
FAIL (2025-07-09 03:03:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.30311284s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
36 out of 60 is failed (threshold=3) [XXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X......] duration:
2.889032605s
FAIL (2025-07-09 03:03:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.889063058s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.XX..X........X.....XXX.X.XXX....X..X.X.X.X.................] duration:
2.808615855s
FAIL (2025-07-09 03:03:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.808719282s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......X...................................X................] duration:
2.306603191s
FAIL (2025-07-09 03:02:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.306637467s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration:
2.353170035s
FAIL (2025-07-09 03:02:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.35321004s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
37 out of 60 is failed (threshold=3) [XXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X.....] duration:
2.819403149s
FAIL (2025-07-09 03:02:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.819442887s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..XX..X........X.....XXX.X.XXX....X..X.X.X.X................] duration:
2.620175882s
FAIL (2025-07-09 03:02:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.620200371s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X.X.................................] duration:
2.48518977s
FAIL (2025-07-09 03:02:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.485246201s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........X...................................X...............] duration:
2.467050433s
FAIL (2025-07-09 03:02:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.467166579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
38 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X....] duration:
2.969614243s
FAIL (2025-07-09 03:02:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.969639533s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X.X................................] duration:
2.403097151s
FAIL (2025-07-09 03:01:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.403131155s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...XX..X........X.....XXX.X.XXX....X..X.X.X.X...............] duration:
2.779311159s
FAIL (2025-07-09 03:01:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.779334556s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
39 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X...] duration:
3.369074465s
FAIL (2025-07-09 03:01:07 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.369223141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration:
2.403317757s
FAIL (2025-07-09 03:01:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.403370855s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X...................................X..............] duration:
2.372064178s
FAIL (2025-07-09 03:01:00 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ting upload
Jul 9 02:48:13.623048621 (377h37m13.506462248s) [upload=301971]: Uploading segments...
Jul 9 02:48:13.623340323 (377h37m13.506753941s) [upload=301971]: Got next segment. Inline: false
Jul 9 02:48:13.623390150 (377h37m13.506803776s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:48:13.623395504 (377h37m13.50680913s) [upload=301971]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:48:13.817252748 (377h37m13.700666374s) [seg_pos={0 0}, upload=301971]: Begin upload segment...
Jul 9 02:48:13.817328879 (377h37m13.700742516s) [seg_pos={0 0}, upload=301971]: Done begin upload segment.
Jul 9 02:48:13.817359351 (377h37m13.700772986s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817380997 (377h37m13.700794628s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817823520 (377h37m13.701237149s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.817891283 (377h37m13.701304922s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818242454 (377h37m13.701656085s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818322245 (377h37m13.701735871s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818545897 (377h37m13.701959539s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818709190 (377h37m13.702122816s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.818886009 (377h37m13.702299638s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.819063767 (377h37m13.702477394s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822137924 (377h37m13.705551558s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822570534 (377h37m13.705984166s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.822930688 (377h37m13.706344317s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823225875 (377h37m13.706639508s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823562888 (377h37m13.706976522s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.823880672 (377h37m13.707294311s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824210482 (377h37m13.707624099s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:13.824584706 (377h37m13.707998336s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Uploading piece...
Jul 9 02:48:14.400917506 (377h37m14.284331136s) [upload=301971]: Next returned nil segment
Jul 9 02:48:14.400948865 (377h37m14.284362476s) [upload=301971]: Waiting for error group managing segments...
Jul 9 02:48:14.539683847 (377h37m14.423097475s) [noise=true , address=77.90.31.14:4679 , piece_id=B5WKU3LFJC4RX5BBGISCGHGDVY2FCRUYK3LHSJPFEYIF7LSAT5NA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.545869119 (377h37m14.429282752s) [noise=true , address=77.90.31.16:4758 , piece_id=E74XD4JFGC4MQXHZ4HWJ35RTVXSZL7XKNG7D2YSGHW3FD6NRC4DQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549093850 (377h37m14.432507475s) [noise=true , address=77.90.31.11:4556 , piece_id=PEV6YNDVYJ6F5ZXBLUGB5K5MXPZ4B6YISCKM44Z7DWQ3L5GLTR2Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.549682016 (377h37m14.433095629s) [noise=true , address=77.90.31.15:4731 , piece_id=SBKWBYURU7VZDMU3OP36HXOOA2SCXHEHUUUVJXFTQFDM6FGD3QYA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559178891 (377h37m14.442592506s) [noise=true , address=77.90.31.19:4852 , piece_id=VA3E5FIKHLWK6O3I5PPDJS52MKB77JC733CVQ46WFLAGIJDVGDZA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.559277031 (377h37m14.442690646s) [noise=true , address=77.90.31.18:4832 , piece_id=DKEOHPYKXWZLJHPFZCERN7KVRY5AQYGZEV5YVW24PPHFKRGY7RAQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.561727156 (377h37m14.445140769s) [noise=true , address=77.90.31.10:4528 , piece_id=Z45WKOXUE65MKNUTAFEBOUUDNCVWPRGI7D6SPOZX7F6DVZXLQI5A, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.579411062 (377h37m14.462824682s) [noise=true , address=77.90.31.13:4653 , piece_id=7K7IYONUOVWCSQMCOZMJ4OENYB3A24WXDQJQ3YFD6X7KZRTHEQMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:14.822904625 (377h37m14.706318242s) [noise=true , address=156.146.43.225:4449 , piece_id=H4TXOQLWVMT6CLQMMFKXP2QLPDIPVUVE2TRMGIXJX3JZWHQJUN3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:15.096350399 (377h37m14.979764016s) [noise=true , address=79.127.205.235:4662 , piece_id=IBJFBZYO3LSKLWJLZROD2CIB7YQXTTOVYRVUKXXODKFH7AT2CEGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:29.510157364 (377h37m29.393570992s) [noise=true , address=104.36.82.2:4763 , piece_id=ITBKWOMUINXGYF3QSKL3EA646UIBTGBZ5K25MZAWC63WYY4HVNEQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:39.632164275 (377h37m39.515577891s) [noise=true , address=104.36.82.12:4161 , piece_id=GAC4GFKQXQIPTEI2VPGAABDI2AYV7GZSXN4L7LY75KW2R6BXKRCQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:48:44.469621505 (377h37m44.35303512s) [noise=true , address=104.36.82.19:4431 , piece_id=XR2KRXZS7VEII53QLWH3NP6RHZ3CL653JPNUWST4EWPOS2ZP2W3Q, seg_pos={0 0}, upload=301971]: Done uploading piece. err: <nil>
Jul 9 02:49:13.355633183 (377h38m13.23904681s) [noise=true , address=104.36.82.44:4885 , piece_id=QWKTSS4453IJRCPI6BCY4OYUMEYQDCHI7HP45HWHPHJOQN5L4INA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355804399 (377h38m13.239218012s) [noise=true , address=104.36.82.35:4598 , piece_id=UW7DOPZFL6TDXN4DQGL6OMEWGVF76SWE5QYB7TQFS4DL7NPAUGGA, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355849615 (377h38m13.239263241s) [noise=true , address=104.36.82.40:4955 , piece_id=3AO226MG7GQ4VJGTFCQKJV7LADBDR52DHIBDWDKACYQQOQDJLWMQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d, address:104.36.82.40:4955): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355894817 (377h38m13.239308443s) [noise=true , address=104.36.82.50:4037 , piece_id=JSQH5SHZVBZJPVB7W2CPQCX7LOU7GKXM3CBKE7N52BXVXBEF3JTQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:49:13.355988218 (377h38m13.239401844s) [noise=true , address=104.36.82.38:4696 , piece_id=QSUPCFVH2E2AR2AY5AO6X4KBNPTUTYDNM4U53PEGJYWDAPH4BPBQ, seg_pos={0 0}, upload=301971]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47232->104.36.82.38:4696: use of closed network connection
Jul 9 02:49:13.356007346 (377h38m13.239420978s) [seg_pos={0 0}, upload=301971]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:49:13.464380779 (377h38m13.347794407s) [upload=301971]: Done uploading segments...
Jul 9 02:49:13.464449732 (377h38m13.347863348s) [upload=301971]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.372098065s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration:
2.478587042s
FAIL (2025-07-09 03:00:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
s) [upload=301814]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:36:36.095816711 (377h25m35.979230349s) [upload=301814]: Done uploading segments...
Jul 9 02:36:36.095861438 (377h25m35.979275066s) [upload=301814]: Upload finished. err: <nil>
Jul 9 02:37:33.550348548 (377h26m33.433762165s) [upload=301826]: Starting upload
Jul 9 02:37:33.550443958 (377h26m33.433857587s) [upload=301826]: Done starting upload
Jul 9 02:37:33.550648388 (377h26m33.434062029s) [upload=301826]: Uploading segments...
Jul 9 02:37:33.550844663 (377h26m33.434258301s) [upload=301826]: Got next segment. Inline: false
Jul 9 02:37:33.550917404 (377h26m33.434331039s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:37:33.550924691 (377h26m33.43433833s) [upload=301826]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:37:33.815909059 (377h26m33.699322692s) [seg_pos={0 0}, upload=301826]: Begin upload segment...
Jul 9 02:37:33.815994368 (377h26m33.699407987s) [seg_pos={0 0}, upload=301826]: Done begin upload segment.
Jul 9 02:37:33.816126929 (377h26m33.699540556s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816186526 (377h26m33.699600158s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.816695002 (377h26m33.700108634s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819297193 (377h26m33.702710831s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.819764828 (377h26m33.703178443s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.820656075 (377h26m33.704069712s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.822922259 (377h26m33.706335899s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.825191080 (377h26m33.708604717s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.826521734 (377h26m33.709935368s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.827123114 (377h26m33.710536745s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829205661 (377h26m33.712619281s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.829794527 (377h26m33.713208161s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.830988926 (377h26m33.714402566s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.831745725 (377h26m33.715159359s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.833607194 (377h26m33.717020827s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:33.834303078 (377h26m33.717716712s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Uploading piece...
Jul 9 02:37:34.434801783 (377h26m34.318215418s) [upload=301826]: Next returned nil segment
Jul 9 02:37:34.434824751 (377h26m34.318238389s) [upload=301826]: Waiting for error group managing segments...
Jul 9 02:37:34.571376185 (377h26m34.454789803s) [noise=true , address=45.91.116.13:4153 , piece_id=Z6XVAGF5J4X7SS6JZ3B5SQEZC5WIGELGIZPGBMFTO6GXDCWQCDQQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.571556166 (377h26m34.454969779s) [noise=true , address=104.36.82.9:4006 , piece_id=VYX4ZOBKJDOBPDDU6XXBLMY7CBYFXHUCFCQ6HSVFOJM674P6ZWTQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.578565129 (377h26m34.461978764s) [noise=true , address=45.91.116.6:4974 , piece_id=F452K44DF3QDGFZS4BCZIFE5L3K6MLGHID242BT2CVK5FIAH5L6A, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.588445961 (377h26m34.471859595s) [noise=true , address=45.91.116.10:4083 , piece_id=RPNK2JI6GXPAYAJ56AXMQSFHW3WNCMYRAKWOLJM7PZOOLFEL34HA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590018122 (377h26m34.473431749s) [noise=true , address=77.90.31.13:4626 , piece_id=IK2IS6NTA74ZALIYY2SOCB2KBCMUXYLMYXCPFJFUYLKCNDSLBHFA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.590534120 (377h26m34.473947751s) [noise=true , address=45.91.116.7:4009 , piece_id=I4IBQHFN6RUJOQP6253J6VUSVXZEXXOCXSMB4E5XZDTNE4QB26KA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.593380073 (377h26m34.476793687s) [noise=true , address=77.90.31.19:4841 , piece_id=P2URXIABU2ZJ2RDXPTYWUDHDVKGF7MLA2J32CTAPGB4YDU3457ZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.595785926 (377h26m34.479199551s) [noise=true , address=77.90.31.21:4879 , piece_id=YMM726IO7X3OPX6YOY3X3F6XYB6L3PSMIFO67OVJVRGPCYBK2LVA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.605401892 (377h26m34.488815504s) [noise=true , address=77.90.31.19:4842 , piece_id=YC2ON4JKTBTHVO42CPCQMFHF6JTVSDV4RVZEXL7NFNIFPUZEIPXQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:34.705694279 (377h26m34.589107907s) [noise=true , address=45.91.116.4:4938 , piece_id=FGYBHC7J2MUWXR6RQ7ZKM4ZZFU75T6SQWCXUPPDX4PL7JSDDAGQA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:37:38.384942197 (377h26m38.268355813s) [noise=true , address=45.91.116.7:4010 , piece_id=QJYX72LLDYA6MGLTN5O6HGKU74KTDSFI3V2MBCBWGOWJQSK7SWKA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: <nil>
Jul 9 02:38:03.210659066 (377h27m3.094072698s) [noise=true , address=104.36.82.5:4838 , piece_id=HZHRW7OZTVDXA2T4Q3SSJAAKW4MFQOSDPNZ6XTH7LR3HGZWWSCZA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210852696 (377h27m3.094266326s) [noise=true , address=104.36.82.8:4970 , piece_id=TNZJWTR4TMUHC26ADZVZJNPWRVWCP4MIICA53FG7KV7YJHOZ4QSA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:38:03.210986553 (377h27m3.094400184s) [noise=true , address=104.36.82.2:4730 , piece_id=3WOQE32LZN3Q7EVPBW5SXP33U2APCP5OFCNIBMS7V7CIBFVWU4IQ, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47666->104.36.82.2:4730: use of closed network connection
Jul 9 02:38:03.211067271 (377h27m3.094480902s) [noise=true , address=104.36.82.4:4803 , piece_id=4UZ44IPO2IKLLB2T3EGUIWKFP6C67QTZRA5D2KP6RNHFO2P2BGHA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48930->104.36.82.4:4803: use of closed network connection
Jul 9 02:38:03.211166349 (377h27m3.094579979s) [noise=true , address=104.36.82.4:4801 , piece_id=LIWPQRP7QIHNDGI2VYMGOBV5LHSFDNBZLH6T3YV34VGQCLA2AMEA, seg_pos={0 0}, upload=301826]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41238->104.36.82.4:4801: use of closed network connection
Jul 9 02:38:03.211187224 (377h27m3.094600859s) [seg_pos={0 0}, upload=301826]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 02:38:03.323304657 (377h27m3.206718295s) [upload=301826]: Done uploading segments...
Jul 9 02:38:03.323358440 (377h27m3.206772072s) [upload=301826]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.478627122s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....XX..X........X.....XXX.X.XXX....X..X.X.X.X..............] duration:
2.794521831s
FAIL (2025-07-09 03:00:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.794550558s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
39 out of 60 is failed (threshold=3) [.XXXXXXXXX.XXXXX..XXX.XX.XX.X.XXXX..X.XXX..X.XX.X.XXXX...X..] duration:
3.481589043s
FAIL (2025-07-09 03:00:22 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.481842905s
core
uplink
us-select-1
us1-placement-20
|