Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.494627476s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:37:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.49467238s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
1.695776538s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:37:27 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.695803563s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
3.576619649s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:37:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.576645925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.68686673s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:37:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.686900316s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.642390766s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:37:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.64242176s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
1.633538625s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:36:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.633559121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
2.465834856s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:36:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.465861184s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.XX.......................XXXXXXX....XX.....................] duration:
5.622437476s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:36:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.622757055s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.825830238s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:36:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.825877378s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.474367802s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:36:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.474390593s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
2.454943001s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:35:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.45504426s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
1.598203478s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:35:27 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.598224602s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
2.392759983s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:35:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.392785903s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..XX.......................XXXXXXX....XX....................] duration:
5.87412467s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:35:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.87415216s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
3.028694461s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:35:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.028729596s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
2.653618295s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:34:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.653647001s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
3.069840795s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:34:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.069872224s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...XX.......................XXXXXXX....XX...................] duration:
3.441389825s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:34:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.441486928s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
3.007339385s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:34:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.007362725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
2.887174113s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:34:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.887197527s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
1.687523213s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:34:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.68759194s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
2.526856987s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:33:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.526885651s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
1.693480123s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:33:45 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.693537142s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....XX.......................XXXXXXX....XX..................] duration:
3.377205567s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:33:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.377231315s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.492622791s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:33:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.492656251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
2.828322938s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:33:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.828379222s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
1.665427035s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:33:19 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.66546897s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
2.76227671s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:33:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.762309267s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
1.658777101s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:32:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.65880521s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
3.145150305s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:32:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 3.145183741s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
1.998140884s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:32:21 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.998161941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.....XX.......................XXXXXXX....XX.................] duration:
3.012607355s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:32:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.012634071s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
3.029675806s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:32:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.029743035s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [......XX.......................XXXXXXX....XX................] duration:
18.032968905s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:32:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.033487658s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration:
2.662645239s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:31:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.662677478s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
1.793033808s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:31:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.793057335s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
3.231840708s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:31:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.231869356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.......XX.......................XXXXXXX....XX...............] duration:
4.103670545s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:31:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.103706696s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......XX....................................................] duration:
2.497478839s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:31:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.497514529s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.727708759s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:31:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.727742459s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.343049787s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:31:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.343072183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [........XX.......................XXXXXXX....XX..............] duration:
8.324265196s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:30:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.324291506s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
2.972850423s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:30:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.972867854s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
1.625090321s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:30:40 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.625108892s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
2.6749368s trace:
https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a
FAIL (2025-10-31 22:30:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload
Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload
Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments...
Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false
Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment...
Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment.
Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece...
Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment
Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments...
Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil>
Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection
Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection
Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection
Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection
Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments...
Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.674966217s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.........XX.......................XXXXXXX....XX.............] duration:
18.069332443s trace:
https://jaeger.dev.storj.io/trace/4e982d41932a4bae
FAIL (2025-10-31 22:30:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.069363546s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
2.54947738s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:30:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.549502273s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
2.743895594s trace:
https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba
FAIL (2025-10-31 22:29:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece...
Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment
Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments...
Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil>
Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection
Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection
Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection
Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection
Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection
Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection
Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection
Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection
Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection
Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments...
Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 2.743981882s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.901433752s trace:
https://jaeger.dev.storj.io/trace/627c68530bc5c078
FAIL (2025-10-31 22:29:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.901445156s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
2.250040733s trace:
https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3
FAIL (2025-10-31 22:29:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.250065166s
core
uplink
us-select-1
us1-placement-20
|