Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
3.215510576s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:22:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.215545838s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...................XX.......................XXXXXXX....XX...] duration:
6.667136877s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:21:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.667309199s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.869880201s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:21:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.870105345s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....................XX.......................XXXXXXX....XX..] duration:
22.715095787s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:21:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.715118147s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration:
2.46382701s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:21:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.463864734s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................X...................................] duration:
2.204344915s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:21:29 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.204389139s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.265839294s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:21:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.265888358s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
3.817189686s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:21:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.817217126s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.....................XX.......................XXXXXXX....XX.] duration:
15.833965676s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:21:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.833983329s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................X..................................] duration:
2.062478018s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:21:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.063192353s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................XX....................................] duration:
3.303615641s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:20:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.303635509s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [......................XX.......................XXXXXXX....XX] duration:
30.000903781s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:20:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000914488s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
3.239815972s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:20:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.239839704s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................XX...................................] duration:
7.693528058s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:20:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.693558414s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.......................XX.......................XXXXXXX....X] duration:
30.00170802s trace:
https://jaeger.dev.storj.io/trace/5a3f1c5efae3d137
FAIL (2025-10-31 22:20:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001736365s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
8.667837819s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:20:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.667866779s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................X.................................] duration:
2.282909103s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:20:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.283012985s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
3.142227074s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:19:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.142251848s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................XX..................................] duration:
2.385481897s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:19:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.385524876s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................X................................] duration:
1.794927834s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:19:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.794981389s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [........................XX.......................XXXXXXX....] duration:
15.528363257s trace:
https://jaeger.dev.storj.io/trace/40ae32090810b3fb
FAIL (2025-10-31 22:19:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.528396212s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
4.864245306s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:19:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.864278949s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
3.354680789s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:19:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.354708712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............................X...............................] duration:
1.791866342s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:18:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.79189602s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.547061103s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:18:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.547084076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.........................XX.......................XXXXXXX...] duration:
13.284590993s trace:
https://jaeger.dev.storj.io/trace/40ae32090810b3fb
FAIL (2025-10-31 22:18:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.284620069s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
3.381290044s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:18:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.381309835s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
3.465719289s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:18:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.465745127s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............................X..............................] duration:
1.949693363s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:18:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.949717221s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.076408066s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:18:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.076430756s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
3.800173068s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:17:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.800201377s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................X.............................] duration:
3.595371073s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:17:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.595400001s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........................XX.......................XXXXXXX..] duration:
5.31819229s trace:
https://jaeger.dev.storj.io/trace/40ae32090810b3fb
FAIL (2025-10-31 22:17:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.318215935s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.594114528s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:17:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.594140728s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................XX...............................] duration:
2.679160896s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:17:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.679188507s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
3.113236057s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:17:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.113315481s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...........................XX.......................XXXXXXX.] duration:
12.896679712s trace:
https://jaeger.dev.storj.io/trace/40ae32090810b3fb
FAIL (2025-10-31 22:17:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.896706959s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.272293954s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:16:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.272328614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................XX..............................] duration:
7.582467436s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:16:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.582500009s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
9.463714441s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:16:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.46374011s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
4.124999669s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:16:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 4.125025261s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................XX.............................] duration:
5.734221209s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:16:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.734249438s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............................X............................] duration:
2.522074384s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:16:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.522106556s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [............................XX.......................XXXXXXX] duration:
30.000896528s trace:
https://jaeger.dev.storj.io/trace/40ae32090810b3fb
FAIL (2025-10-31 22:16:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000908394s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................XX............................] duration:
3.106252747s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:16:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.106276571s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................X...........................] duration:
3.249652748s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:16:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.249695682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
20.452147994s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:16:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.452170851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
3.055327714s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:15:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.055356071s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............................XX.......................XXXXXX] duration:
30.001015422s trace:
https://jaeger.dev.storj.io/trace/7bae5e2714fdbabb
FAIL (2025-10-31 22:15:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001038175s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................X..........................] duration:
2.516036647s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:15:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.516064348s
core
uplink
us-select-1
us1-placement-20
|