Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 190703 reports (Page 441 of 3815)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................XX.....XX...............................X] duration: 1m0.112414505s trace: https://jaeger.dev.storj.io/trace/173d8e4dd9dfc2ba FAIL (2025-10-31 21:54:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Uploading piece... Oct 31 21:53:44.862904010 (34h12m47.564496444s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Uploading piece... Oct 31 21:53:44.863070435 (34h12m47.5646629s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Uploading piece... Oct 31 21:53:44.863347140 (34h12m47.564939594s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Uploading piece... Oct 31 21:53:44.863580564 (34h12m47.565173019s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Uploading piece... Oct 31 21:53:44.863769233 (34h12m47.565361686s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Uploading piece... Oct 31 21:53:44.864085322 (34h12m47.565677776s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Uploading piece... Oct 31 21:53:45.422630814 (34h12m48.124223262s) [upload=30100]: Next returned nil segment Oct 31 21:53:45.422656839 (34h12m48.124249286s) [upload=30100]: Waiting for error group managing segments... Oct 31 21:53:45.585092068 (34h12m48.286684516s) [noise=true , address=145.79.227.51:4409 , piece_id=7TC7P7YCVK5VHPRGUC5XXABIQON6JIODVRPZZ6BBTQNOPZHEBHQA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil> Oct 31 21:53:45.587483046 (34h12m48.289075489s) [noise=true , address=145.79.227.44:4222 , piece_id=NS7RAW7JXHUL3OHBGRMHLULNTEFXUFYVC3XGMEOU7CRDT4A7JUGQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil> Oct 31 21:53:45.590833383 (34h12m48.292425831s) [noise=true , address=145.79.227.17:4983 , piece_id=IXEP3EFLE7D564KVMKJYAMINIZDWMXQKNG3EVQ7I4HVQOLWDCPOQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil> Oct 31 21:53:45.595886856 (34h12m48.297479304s) [noise=true , address=145.79.227.7:4321 , piece_id=5N6LYRX6634XWOL7QJSY7TLQR4UBSSE2ZI5Z4CNG5BRQJFQTN2QQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil> Oct 31 21:53:45.599860574 (34h12m48.301453022s) [noise=true , address=145.79.227.3:4169 , piece_id=7SK6IOMYVQGFZBCX5FQBHMNLYHT53FIDKTYGDUNRZJHLLCYC7J4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil> Oct 31 21:53:45.611889778 (34h12m48.313482226s) [noise=true , address=145.79.227.6:4273 , piece_id=EAPVETINN2RFY7NJWX76DOB75QCDEKSJ56IDUHK5KUYNQHUWDLPA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: <nil> Oct 31 21:54:44.478981735 (34h13m47.180574194s) [noise=true , address=45.91.116.10:4102 , piece_id=FR3ELN5LB3JOPBHHEXKI2SBJHSAJHU2YWY442OPJ3QKMUOS23LXA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12AXruuVjtSTQ6AjVR4xV6KRWQbpw6X7CgH5aBU71LX53eNghc1, address:45.91.116.10:4102): protocol: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54788->45.91.116.10:4102: use of closed network connection Oct 31 21:54:44.479047899 (34h13m47.180640358s) [noise=true , address=77.90.31.13:4660 , piece_id=JAMES4YM4YDZGHKDHSZBPX5KD3FRXZKHBBPG6XYGZRUKQ5H7J5LQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35622->77.90.31.13:4660: use of closed network connection Oct 31 21:54:44.479195440 (34h13m47.180787904s) [noise=true , address=77.90.31.14:4673 , piece_id=YSDLCXJMTD6KXQQXNT4OUQSYZXBCONXMAB5ID76MU6WFGNNBMKGA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39438->77.90.31.14:4673: use of closed network connection Oct 31 21:54:44.479222980 (34h13m47.18081544s) [noise=true , address=77.90.31.19:4859 , piece_id=2APT4MM3EKJIHVBQU6FVIW4QJDIM2PMI5RO2TZCK426WBA7ML3BA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:44.479244364 (34h13m47.180836822s) [noise=true , address=77.90.31.16:4746 , piece_id=7ZHHVJG4GHB5N23QZITZ4OK4TQKSHILCFODX7NGMUGN755EKAY6Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60782->77.90.31.16:4746: use of closed network connection Oct 31 21:54:44.479444439 (34h13m47.1810369s) [noise=true , address=77.90.31.12:4610 , piece_id=4PP2DKH72QX4LCD5P3AG45NA3GQAH3QQK46OL6ND5I2ABJVQT5QA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: read tcp 139.59.144.22:54388->77.90.31.12:4610: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:44.479579545 (34h13m47.181172004s) [noise=true , address=45.91.116.4:4970 , piece_id=PMHG3YAWDSP2TLHIPHZ5JBHS2RM4HKPCEK5KQ5PUO7PW44BAVM4Q, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56156->45.91.116.4:4970: use of closed network connection Oct 31 21:54:44.479651050 (34h13m47.181243509s) [noise=true , address=77.90.31.15:4708 , piece_id=3LKOJCM6D3LXDY3B7VIVDYVWKXI7YWEIQUPFGXBFAPCZBP7HAYTQ, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, address:77.90.31.15:4708): protocol: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37378->77.90.31.15:4708: use of closed network connection Oct 31 21:54:44.479704965 (34h13m47.181297423s) [noise=true , address=45.91.116.13:4167 , piece_id=FH6EWM5PPZB35L7ZSNDH5H42AGYOASN7IEPHJYON3IXQWWKGHFDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp, address:45.91.116.13:4167): protocol: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48568->45.91.116.13:4167: use of closed network connection Oct 31 21:54:44.479748167 (34h13m47.181340849s) [noise=true , address=45.91.116.6:4003 , piece_id=BBO3WKL4ZFAAXFRWFMR4O5ODJZ2WNVHDYHLJAUR3YBGYUD34UWOA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF, address:45.91.116.6:4003): protocol: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54596->45.91.116.6:4003: use of closed network connection Oct 31 21:54:44.479796299 (34h13m47.181388758s) [noise=true , address=77.90.31.18:4814 , piece_id=MNTMLG2OZJWVVJNNFJTXP5L2PSVR566XNBMMWC73MFJKIY4ZQZVA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:54:44.479836852 (34h13m47.181429313s) [noise=true , address=77.90.31.10:4528 , piece_id=6RAGWC7DSKWXLSCAMVN6S3ATREQ3YTU44RKS3DE6CSRWRJER3PDA, seg_pos={0 0}, upload=30100]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49108->77.90.31.10:4528: use of closed network connection Oct 31 21:54:44.479865024 (34h13m47.181457476s) [seg_pos={0 0}, upload=30100]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Oct 31 21:54:44.588459586 (34h13m47.290052034s) [upload=30100]: Done uploading segments... Oct 31 21:54:44.588493656 (34h13m47.290086107s) [upload=30100]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.112438936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.........X.......X...................................X] duration: 5.000934109s trace: https://jaeger.dev.storj.io/trace/2e6b18fc118d87e3 FAIL (2025-10-31 21:54:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000958605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.....X.X..................................XX] duration: 30.110986847s trace: https://jaeger.dev.storj.io/trace/2aa667c5bfc9b01a FAIL (2025-10-31 21:54:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:54:11.121982241 (34h13m13.823574706s) [upload=30104]: Starting upload Oct 31 21:54:11.122136701 (34h13m13.823729136s) [upload=30104]: Done starting upload Oct 31 21:54:11.122186892 (34h13m13.823779332s) [upload=30104]: Uploading segments... Oct 31 21:54:11.122620713 (34h13m13.824213162s) [upload=30104]: Got next segment. Inline: false Oct 31 21:54:11.122725023 (34h13m13.82431748s) [upload=30104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:54:11.122732342 (34h13m13.824324794s) [upload=30104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:54:11.238493129 (34h13m13.940085589s) [seg_pos={0 0}, upload=30104]: Begin upload segment... Oct 31 21:54:11.238554433 (34h13m13.940146893s) [seg_pos={0 0}, upload=30104]: Done begin upload segment. Oct 31 21:54:11.238620315 (34h13m13.940212775s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.238660365 (34h13m13.940252817s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.239235465 (34h13m13.940827926s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.239392545 (34h13m13.940984985s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.239853883 (34h13m13.941446344s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.240230893 (34h13m13.941823343s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.240782453 (34h13m13.942374914s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.241233578 (34h13m13.942826041s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.241686583 (34h13m13.943279039s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.242256764 (34h13m13.943849225s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.242462416 (34h13m13.944054877s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.243068388 (34h13m13.94466085s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.243618759 (34h13m13.945211218s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.244111210 (34h13m13.945703672s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.244688249 (34h13m13.946280711s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.245449941 (34h13m13.9470424s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Uploading piece... Oct 31 21:54:11.848473657 (34h13m14.550066109s) [upload=30104]: Next returned nil segment Oct 31 21:54:11.848489334 (34h13m14.550081793s) [upload=30104]: Waiting for error group managing segments... Oct 31 21:54:11.986106978 (34h13m14.687699433s) [noise=true , address=145.79.227.4:4185 , piece_id=V7ABVFRGSKLLDLR54XCCSEAGDHAE6UGVUX5RMXDAAUE6EKTYTXOA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.987949590 (34h13m14.689542038s) [noise=true , address=145.79.227.20:4437 , piece_id=JR5WI2UZTSEUKWBZDQQWUSK3UAMEVGCQS62BNZCTQSVWF2LZKDYA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.988038515 (34h13m14.689630963s) [noise=true , address=145.79.227.11:4731 , piece_id=UCQ7CM62LZAITSMDOPIEWWTXRV53VC4YEULA6VAZVGQNYSLQKEBQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.988992745 (34h13m14.690585194s) [noise=true , address=145.79.227.8:4331 , piece_id=WCVBXOQ2266JZDR2G4T3MD5VXRTILJXOT34GR46DO3YX7H6RUICA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.990180545 (34h13m14.691772996s) [noise=true , address=145.79.227.15:4875 , piece_id=N5AKELZ6YIQMVMTPK3QXYSTNLPOITKYU52LH3HJGQXO57JO4FYNA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.990495447 (34h13m14.692087897s) [noise=true , address=145.79.227.21:4475 , piece_id=24C5IMUPHY2ILCPLXTAP5VJ7XSVTKVODXAXP3W4PC43GTL3DAOUA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.990892083 (34h13m14.692484532s) [noise=true , address=145.79.227.30:4762 , piece_id=J2RAZQDTVADZTJPZK7EW44VYOFRTXF5XGHUMCOKYBIZTBEMNG7WQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.991595571 (34h13m14.693188021s) [noise=true , address=145.79.227.32:4836 , piece_id=WFB2WUSLZQUYSH2X7BYUCFFRKCUAE6L2PJC4KWWU5PF2MQIXSM7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.995653337 (34h13m14.697245787s) [noise=true , address=145.79.227.27:4689 , piece_id=KTARCTO74WRCQYRRKWIIYHD2KDJUQBTS7VLO43ZVIRULU5M27E7Q, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:11.997059753 (34h13m14.698652205s) [noise=true , address=145.79.227.27:4690 , piece_id=JP3XKDRGHCXFEUFL2PWZH47FI3RL6Z32ABHFVJTAWORQLXVTRZEA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:12.000362536 (34h13m14.701955022s) [noise=true , address=145.79.227.31:4798 , piece_id=J3GN6R2IO2ZLW2MM5PKTDNACYYUWXUVK3Q2YI77XW4XAL2CNWD2A, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:12.000772309 (34h13m14.702364789s) [noise=true , address=145.79.227.32:4835 , piece_id=OQY6TVSC5BVHFISCE4MIBDHGZLKAB6MJ4MWLQ6NC2GAA2TPG5RVQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: <nil> Oct 31 21:54:40.831183505 (34h13m43.532775955s) [noise=true , address=77.90.31.12:4589 , piece_id=NLFGM6HWNPGZN7VXUXPJCTS5HHUYKGXFKFPUTMDBAWYHFZNJXLDA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38376->77.90.31.12:4589: use of closed network connection Oct 31 21:54:40.831316010 (34h13m43.53290846s) [noise=true , address=45.91.116.14:4190 , piece_id=Z54UQDTS2AUU3EB7KWP2TWB3SWX3DEEPFMBVMOCCILEUSYCXJ3WA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51184->45.91.116.14:4190: use of closed network connection Oct 31 21:54:40.831462898 (34h13m43.533055338s) [noise=true , address=77.90.31.19:4843 , piece_id=WGCUFJCLPOQQC2VERKTMDT7RILBE3ZN5KFBCXPAEUTQO5BUVCQLA, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37526->77.90.31.19:4843: use of closed network connection Oct 31 21:54:40.831508926 (34h13m43.533101361s) [noise=true , address=45.91.116.4:4939 , piece_id=IZCMM3XW7CEDCDYJWCJYFGKQLPG73LUOVPYWEX6XWD5TDM7D5OYQ, seg_pos={0 0}, upload=30104]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46036->45.91.116.4:4939: use of closed network connection Oct 31 21:54:40.831539625 (34h13m43.533132085s) [seg_pos={0 0}, upload=30104]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:40.939548401 (34h13m43.641140862s) [upload=30104]: Done uploading segments... Oct 31 21:54:40.939621028 (34h13m43.641213486s) [upload=30104]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.111002048s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X.........X.......X...................................] duration: 2.928333806s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:54:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.92835411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.....X.X..................................X] duration: 30.109395471s trace: https://jaeger.dev.storj.io/trace/209bbd7384ed5ff6 FAIL (2025-10-31 21:54:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 695 (34h12m43.560158152s) [upload=30099]: Done starting upload Oct 31 21:53:40.858777730 (34h12m43.56037017s) [upload=30099]: Uploading segments... Oct 31 21:53:40.859352397 (34h12m43.560944855s) [upload=30099]: Got next segment. Inline: false Oct 31 21:53:40.859441803 (34h12m43.561034368s) [upload=30099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:40.859449726 (34h12m43.561042183s) [upload=30099]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:40.973088562 (34h12m43.674681022s) [seg_pos={0 0}, upload=30099]: Begin upload segment... Oct 31 21:53:40.973153595 (34h12m43.674746053s) [seg_pos={0 0}, upload=30099]: Done begin upload segment. Oct 31 21:53:40.973194574 (34h12m43.67478703s) [noise=true , address=145.79.227.25:4618 , piece_id=VBLVQPMHFXFSBE4FROSCLC5XB5ITXCPT45XYHM56OTUHIP4N5S7Q, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.973227363 (34h12m43.674819815s) [noise=true , address=145.79.227.24:4583 , piece_id=EBV5QMLSF72QHTKL2QESQTL4YVAAABCTCOZQVGFVTOK2UBPHDCMQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.973862655 (34h12m43.675455114s) [noise=true , address=145.79.227.27:4691 , piece_id=ZNYKH4AF5GXTPPTAAKDUWLSSZGWROEQVGTASFMLBUG2ZR3WXUXAA, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.974932641 (34h12m43.6765251s) [noise=true , address=77.90.31.10:4519 , piece_id=V3VA2W72SZIDRFUA7KTU4U6WQWPIVFQ3LVMAD2ZDTHWYXSWK5RSA, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.975174749 (34h12m43.676767204s) [noise=true , address=145.79.227.21:4474 , piece_id=G6IWCTXBZ522OTST5BZTQIAZK6Y3KLOKCV4NVZCIZVEXEZX3B5EQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.975635270 (34h12m43.677227731s) [noise=true , address=145.79.227.13:4804 , piece_id=ZJWFLGKIB7M5ZAGQRSY6M52462JI4AVB2NCRZFF5BRDDG74I6XMQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.975644718 (34h12m43.677237174s) [noise=true , address=45.91.116.14:4190 , piece_id=TQPQHVAGLIEEGIBJZ5LDKHYSNSY2BVZW5O56HT7MYMC2BPM3KRYQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.976050778 (34h12m43.677643229s) [noise=true , address=77.90.31.14:4663 , piece_id=DAOF3THNHNLUQLHK6WO72KUQ7AYHFCQ5XNA22QUO2UA6X5VH6ZPQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.976209847 (34h12m43.677802307s) [noise=true , address=45.91.116.4:4938 , piece_id=PSEU5SZTXNF5ASJYUZR7RQRIUAQ3TUAUMDUIY6S4WP2BFHSFAVEQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.976441962 (34h12m43.678034417s) [noise=true , address=77.90.31.12:4592 , piece_id=5IKDR5PBFIOOKOLQZK2MYIG7LERQCBT3N4JH3BCX3TSJ57RRKU7Q, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.976939210 (34h12m43.678531668s) [noise=true , address=77.90.31.13:4626 , piece_id=6RIXSPQXBHRI6B2J4DZBBGOL53N6M5UXTHEMXLJ42NSEWW7PMPLQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.977671762 (34h12m43.67926422s) [noise=true , address=145.79.227.18:4006 , piece_id=RULIO3KVOBBWXN4FWLX5ZXQXSHBNS25FOHUPC7HWRI6IIYNLPHVA, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.978284269 (34h12m43.679876728s) [noise=true , address=145.79.227.4:4185 , piece_id=JMV2AOKHDTTKTBGHWWGMUA6IL4C6MFG6LM3X45XDKGGLI6NVGQZA, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.979702865 (34h12m43.681295325s) [noise=true , address=145.79.227.15:4873 , piece_id=TXKHV7DAHMSKQI4USWNXRRKNLZBYILGKV2QGJUQ3Y6Y5R4WFIWTA, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.980371212 (34h12m43.681963651s) [noise=true , address=145.79.227.31:4798 , piece_id=QECYJDETXTFLCUK6GHCRNBIALC2DEAUS3KWQZISFU32BGV3MG6UQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:40.983857685 (34h12m43.68545014s) [noise=true , address=145.79.227.20:4439 , piece_id=Q3FI267LCG5CDMHW2YSMAEFVMRR4R4B67UYMM2CSA77KA656QMRQ, seg_pos={0 0}, upload=30099]: Uploading piece... Oct 31 21:53:41.591298155 (34h12m44.292890604s) [upload=30099]: Next returned nil segment Oct 31 21:53:41.591314841 (34h12m44.292907296s) [upload=30099]: Waiting for error group managing segments... Oct 31 21:53:41.731224872 (34h12m44.432817323s) [noise=true , address=145.79.227.31:4798 , piece_id=QECYJDETXTFLCUK6GHCRNBIALC2DEAUS3KWQZISFU32BGV3MG6UQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.732291999 (34h12m44.433884455s) [noise=true , address=145.79.227.18:4006 , piece_id=RULIO3KVOBBWXN4FWLX5ZXQXSHBNS25FOHUPC7HWRI6IIYNLPHVA, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.733333826 (34h12m44.434926277s) [noise=true , address=145.79.227.21:4474 , piece_id=G6IWCTXBZ522OTST5BZTQIAZK6Y3KLOKCV4NVZCIZVEXEZX3B5EQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.734156588 (34h12m44.435749041s) [noise=true , address=145.79.227.20:4439 , piece_id=Q3FI267LCG5CDMHW2YSMAEFVMRR4R4B67UYMM2CSA77KA656QMRQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.734399332 (34h12m44.43599178s) [noise=true , address=145.79.227.27:4691 , piece_id=ZNYKH4AF5GXTPPTAAKDUWLSSZGWROEQVGTASFMLBUG2ZR3WXUXAA, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.734774392 (34h12m44.436366841s) [noise=true , address=145.79.227.13:4804 , piece_id=ZJWFLGKIB7M5ZAGQRSY6M52462JI4AVB2NCRZFF5BRDDG74I6XMQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.734855681 (34h12m44.436448195s) [noise=true , address=145.79.227.25:4618 , piece_id=VBLVQPMHFXFSBE4FROSCLC5XB5ITXCPT45XYHM56OTUHIP4N5S7Q, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.736700593 (34h12m44.438293049s) [noise=true , address=145.79.227.15:4873 , piece_id=TXKHV7DAHMSKQI4USWNXRRKNLZBYILGKV2QGJUQ3Y6Y5R4WFIWTA, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.739461297 (34h12m44.441053746s) [noise=true , address=145.79.227.24:4583 , piece_id=EBV5QMLSF72QHTKL2QESQTL4YVAAABCTCOZQVGFVTOK2UBPHDCMQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:53:41.741364994 (34h12m44.442957443s) [noise=true , address=145.79.227.4:4185 , piece_id=JMV2AOKHDTTKTBGHWWGMUA6IL4C6MFG6LM3X45XDKGGLI6NVGQZA, seg_pos={0 0}, upload=30099]: Done uploading piece. err: <nil> Oct 31 21:54:10.565683501 (34h13m13.267275959s) [noise=true , address=45.91.116.4:4938 , piece_id=PSEU5SZTXNF5ASJYUZR7RQRIUAQ3TUAUMDUIY6S4WP2BFHSFAVEQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: write tcp 139.59.144.22:56588->45.91.116.4:4938: use of closed network connection; write tcp 139.59.144.22:56588->45.91.116.4:4938: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56588->45.91.116.4:4938: use of closed network connection Oct 31 21:54:10.566032076 (34h13m13.267624536s) [noise=true , address=77.90.31.12:4592 , piece_id=5IKDR5PBFIOOKOLQZK2MYIG7LERQCBT3N4JH3BCX3TSJ57RRKU7Q, seg_pos={0 0}, upload=30099]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:37214->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:10.566086843 (34h13m13.267679289s) [noise=true , address=77.90.31.10:4519 , piece_id=V3VA2W72SZIDRFUA7KTU4U6WQWPIVFQ3LVMAD2ZDTHWYXSWK5RSA, seg_pos={0 0}, upload=30099]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 139.59.144.22:33706->77.90.31.10:4519: use of closed network connection; write tcp 139.59.144.22:33706->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33706->77.90.31.10:4519: use of closed network connection Oct 31 21:54:10.566181608 (34h13m13.267774074s) [noise=true , address=45.91.116.14:4190 , piece_id=TQPQHVAGLIEEGIBJZ5LDKHYSNSY2BVZW5O56HT7MYMC2BPM3KRYQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 139.59.144.22:40990->45.91.116.14:4190: use of closed network connection; write tcp 139.59.144.22:40990->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40990->45.91.116.14:4190: use of closed network connection Oct 31 21:54:10.566416457 (34h13m13.268008916s) [noise=true , address=77.90.31.13:4626 , piece_id=6RIXSPQXBHRI6B2J4DZBBGOL53N6M5UXTHEMXLJ42NSEWW7PMPLQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 139.59.144.22:49226->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:10.566527222 (34h13m13.268119774s) [noise=true , address=77.90.31.14:4663 , piece_id=DAOF3THNHNLUQLHK6WO72KUQ7AYHFCQ5XNA22QUO2UA6X5VH6ZPQ, seg_pos={0 0}, upload=30099]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 139.59.144.22:42356->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:10.566558411 (34h13m13.268150861s) [seg_pos={0 0}, upload=30099]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 31 21:54:10.673234001 (34h13m13.374826464s) [upload=30099]: Done uploading segments... Oct 31 21:54:10.673275393 (34h13m13.374867857s) [upload=30099]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.109418523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...........X......XX..................................X] duration: 30.000576978s trace: https://jaeger.dev.storj.io/trace/1596fb12beaa02b4 FAIL (2025-10-31 21:54:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000673661s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X.........X.......X..................................] duration: 1.628378752s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:53:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.628419346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.80610056s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:52:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.806186642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....X.X..................................] duration: 2.909212207s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:52:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.909243834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.857536822s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:52:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.857570567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX.....XX...............................] duration: 2.334236533s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.334260194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........X......XX..................................] duration: 2.772723425s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:52:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.772766894s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X.........X.......X.................................] duration: 1.605789017s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:51:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.605809201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.830640459s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:51:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.830665287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...........X......XX.................................] duration: 2.552052974s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:51:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.552082204s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....X.X.................................] duration: 2.369176559s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:51:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.369206925s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X.........X.......X................................] duration: 1.720805687s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:51:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.720835784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX.....XX..............................] duration: 2.301304016s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:51:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.301329241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.220020622s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:51:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.220046135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........X......XX................................] duration: 2.640245253s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:51:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.640270687s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.....X.X................................] duration: 2.385865349s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.385901664s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...........X......XX...............................] duration: 2.486235704s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:50:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.486262975s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XX.....XX.............................] duration: 2.433478686s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:50:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.433509769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.....X.X...............................] duration: 2.408960534s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:50:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.408990015s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X.........X.......X...............................] duration: 2.277234323s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:50:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.277305718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.501234807s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:50:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501318292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.....X.X..............................] duration: 2.307257302s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.307300545s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........X......XX..............................] duration: 2.687131216s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:50:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.687155065s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.872293851s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:49:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.872333349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X...........X......XX.............................] duration: 2.702920014s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:49:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.702973688s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.....X.X.............................] duration: 2.397510716s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:49:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.397555163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX.....XX............................] duration: 2.390139491s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.390250869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X.........X.......X..............................] duration: 1.729699311s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:49:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.729722759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X...........X......XX............................] duration: 2.552726442s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:49:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.552750661s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.....X.X............................] duration: 2.390506668s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:48:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.390537135s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.982878607s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:48:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.982905845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.....X.X...........................] duration: 2.346776464s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.346802844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XX.....XX...........................] duration: 2.572866373s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:48:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.572921678s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X...........X......XX...........................] duration: 2.889935376s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:48:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.889955928s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.....X.X..........................] duration: 2.321719705s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:48:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.321749027s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X.........X.......X.............................] duration: 1.685707765s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:48:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685737901s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X...........X......XX..........................] duration: 2.912328698s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:48:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.912352608s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX.....XX..........................] duration: 2.565532402s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.565566149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X.........X.......X............................] duration: 2.146107019s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:48:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.146135652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.....X.X.........................] duration: 2.551841973s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.551870765s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.790305356s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:47:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.790331754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X.........X.......X...........................] duration: 1.636207468s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:47:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.636232123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.........................XX.....XX.........................] duration: 2.410870205s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:47:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.410887404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.895052743s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:47:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.89507205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.....X.X........................] duration: 2.976932567s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.976960148s core uplink us-select-1

1
...
439
440
441
442
443
...
3815