Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 46.584565165s FAIL (2025-08-16 15:53:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 46.584597639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X...........X................................] duration: 946.546485ms FAIL (2025-08-16 15:52:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 946.58544ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.................XX..........] duration: 2.70828838s FAIL (2025-08-16 15:52:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.708322114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX...X..........................X......................X.] duration: 31.62869101s FAIL (2025-08-16 15:52:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 31.628746306s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.................XX.........] duration: 24.5126606s FAIL (2025-08-16 15:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 24.51269478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X...........X...............................] duration: 1.067650442s FAIL (2025-08-16 15:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.067677354s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX...X..........................X......................X] duration: 361.703907ms FAIL (2025-08-16 15:51:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 361.727528ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX...X..........................X......................] duration: 17.961352781s FAIL (2025-08-16 15:51:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 17.961393445s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 14.283593681s FAIL (2025-08-16 15:51:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 14.283618838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 26.629848206s FAIL (2025-08-16 15:51:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 26.62987822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX...X..........................X.....................] duration: 36.537203051s FAIL (2025-08-16 15:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 36.537242688s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X...........X..............................] duration: 1.180348339s FAIL (2025-08-16 15:51:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.180394736s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.................XX........] duration: 6.593084801s FAIL (2025-08-16 15:50:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.593136358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 16.998632708s FAIL (2025-08-16 15:50:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 16.998664377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX...X..........................X....................] duration: 39.671139716s FAIL (2025-08-16 15:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 39.671164988s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 11.608226628s FAIL (2025-08-16 15:50:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 11.60826537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X...........X.............................] duration: 1.328105204s FAIL (2025-08-16 15:49:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.328156819s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.................XX.......] duration: 5.546162898s FAIL (2025-08-16 15:49:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.546195419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX...X..........................X...................] duration: 24.657573469s FAIL (2025-08-16 15:49:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 24.65760996s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X...........X............................] duration: 1.496299126s FAIL (2025-08-16 15:49:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.496332546s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.................XX......] duration: 3.304642616s FAIL (2025-08-16 15:49:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 3.304683176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X...........X...........................] duration: 1.390103314s FAIL (2025-08-16 15:49:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.390138662s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 29.878188218s FAIL (2025-08-16 15:49:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 29.878272137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...........X..........................] duration: 1.656153545s FAIL (2025-08-16 15:48:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.65618392s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX...X..........................X..................] duration: 39.857211566s FAIL (2025-08-16 15:48:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 39.857245217s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.................XX.....] duration: 11.48581354s FAIL (2025-08-16 15:48:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.485845833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X...........X.........................] duration: 859.303805ms FAIL (2025-08-16 15:48:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 859.338121ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 15.893260106s FAIL (2025-08-16 15:48:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 15.893292286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.................XX....] duration: 19.251739497s FAIL (2025-08-16 15:47:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 19.251787463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX...X..........................X.................] duration: 52.357691396s FAIL (2025-08-16 15:47:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 52.357716871s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 24.100117996s FAIL (2025-08-16 15:47:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 24.10015153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...........X........................] duration: 1.685794123s FAIL (2025-08-16 15:47:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.68585562s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX...X..........................X................] duration: 48.306194128s FAIL (2025-08-16 15:46:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 48.306214033s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.................XX...] duration: 13.439040968s FAIL (2025-08-16 15:46:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 13.439077083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XX...X..........................X...............] duration: 26.192939337s FAIL (2025-08-16 15:46:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 26.192974008s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X...........X.......................] duration: 1.252980547s FAIL (2025-08-16 15:46:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.253029059s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 23.416493047s FAIL (2025-08-16 15:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 23.416538631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.................XX..] duration: 6.709779026s FAIL (2025-08-16 15:45:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.709813253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 35.277758178s FAIL (2025-08-16 15:45:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 35.277782445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX...X..........................X..............] duration: 45.172235095s FAIL (2025-08-16 15:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 45.172279769s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...........X......................] duration: 1.315069396s FAIL (2025-08-16 15:45:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.315167788s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 43.123653273s FAIL (2025-08-16 15:44:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 43.123698202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.................XX.] duration: 2.445687198s FAIL (2025-08-16 15:44:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.445722616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX...X..........................X.............] duration: 48.23276284s FAIL (2025-08-16 15:44:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 48.232788584s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...........X.....................] duration: 1.159239917s FAIL (2025-08-16 15:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.159268649s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 37.335808038s FAIL (2025-08-16 15:44:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 37.335915962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.................XX] duration: 30.101832985s FAIL (2025-08-16 15:44:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.101861054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.................X] duration: 30.112302876s FAIL (2025-08-16 15:43:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:43:07.622835035 (190h17m29.581932285s) [upload=198084]: Starting upload Aug 16 15:43:07.622942172 (190h17m29.582039419s) [upload=198084]: Done starting upload Aug 16 15:43:07.623031604 (190h17m29.582128854s) [upload=198084]: Uploading segments... Aug 16 15:43:07.623387386 (190h17m29.582484623s) [upload=198084]: Got next segment. Inline: false Aug 16 15:43:07.623441663 (190h17m29.582538918s) [upload=198084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:07.623452950 (190h17m29.582550202s) [upload=198084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:07.734201384 (190h17m29.693298644s) [seg_pos={0 0}, upload=198084]: Begin upload segment... Aug 16 15:43:07.734281124 (190h17m29.693378369s) [seg_pos={0 0}, upload=198084]: Done begin upload segment. Aug 16 15:43:07.734324112 (190h17m29.693421362s) [noise=true , address=45.91.116.7:4011 , piece_id=L52OSPTBUDTGCSUMZIDL2EW54J7AIJYXITXC7QWCHYVN4NYEMT7Q, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.734352807 (190h17m29.693450062s) [noise=true , address=45.91.116.9:4261 , piece_id=4GL2AM3WJCUCTKABDOCKZYRJN32KPYGJPU3NTHOQNOSZ7FGGSWZA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.734801717 (190h17m29.693898968s) [noise=true , address=104.36.82.6:4876 , piece_id=E2CRSYPOHOVTDGDWFF6T4TZLIL7BNYP6B4XI7CR36IKCVAU2AXZQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.734820106 (190h17m29.693917355s) [noise=true , address=45.91.116.5:4297 , piece_id=5632ZHQWXBS66OW4CR5UHEH7DCJOYMRUQS3X37QF7FFOUX3JMJDA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735228202 (190h17m29.694325451s) [noise=true , address=77.90.31.14:4663 , piece_id=TEGACRI7HO27B4ZI3QV62N7SIDSEQ4BE4ZQW77MKYV7O5GGKFJPQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735238542 (190h17m29.694335799s) [noise=true , address=104.36.82.4:4804 , piece_id=3QYWY4AWHJX3U4WBOBQ2OKDBJ3KPC4YTWSKR3UJG2H7QKB3GLZYQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735627561 (190h17m29.694724809s) [noise=true , address=104.36.82.12:4149 , piece_id=QY7DO7YY7R3CG5P3P5UKU25SO6EYYIDB47EEN2MBWOKSWGR3NZ6Q, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735647590 (190h17m29.694744847s) [noise=true , address=104.36.82.2:4731 , piece_id=ZL4OSIWH7TC3OX4JUN6DFOYEKN7QCWJBIJAX4ROLPTDM3SGUDP2A, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.736036851 (190h17m29.695134087s) [noise=true , address=45.91.116.8:4048 , piece_id=J536X5BXF6OERIFMFNYLXR25QHKTWWJ7DPSNSIIP4QXMTYHAUWWA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.736047754 (190h17m29.695145s) [noise=true , address=77.90.31.11:4555 , piece_id=HS5JACTQ5UUE3CAAXLDUDVM6CNVC76QWUT2VQKWFXOF5EL3O2EQA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.736425865 (190h17m29.695523115s) [noise=true , address=104.36.82.11:4115 , piece_id=ZZLZE25XOWBWJDDH7FEEYR74YPCVSBVI7CQD6GTG6CQMEFGHK2KA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.737634028 (190h17m29.696731264s) [noise=true , address=104.36.82.5:4839 , piece_id=P47CLM36SLM6E7UKL5YYRVUCTYMMGE63PHFN6QNQR4GVT325GR5A, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.738181454 (190h17m29.697278704s) [noise=true , address=104.36.82.7:4912 , piece_id=HQSFCGME3FF7TUPOXD4GUJUQEF2Z5YWAZQGOAAZKILZBCGUBCW7Q, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.738601775 (190h17m29.697699024s) [noise=true , address=104.36.82.8:4971 , piece_id=WR6X7OIPS2522GSM6L6JG4OJRAOD5GMEETCJOFHEQCRKZZQROTCQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.738997566 (190h17m29.698094821s) [noise=true , address=45.91.116.6:4974 , piece_id=ZEDZDURBN632VT57V5OQX5HNQE4EPEQPHW5PVRDO535EUDXPXTCQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.740149568 (190h17m29.699246809s) [noise=true , address=77.90.31.18:4807 , piece_id=5HZXGFQK6ZMM6NW6SOT7UKVR5KWA2RKNPJYHG7WYBOK6KZ57JRIA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:08.350731304 (190h17m30.30982855s) [upload=198084]: Next returned nil segment Aug 16 15:43:08.350750069 (190h17m30.309847317s) [upload=198084]: Waiting for error group managing segments... Aug 16 15:43:08.523524057 (190h17m30.482621306s) [noise=true , address=104.36.82.6:4876 , piece_id=E2CRSYPOHOVTDGDWFF6T4TZLIL7BNYP6B4XI7CR36IKCVAU2AXZQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.530214696 (190h17m30.489311951s) [noise=true , address=77.90.31.11:4555 , piece_id=HS5JACTQ5UUE3CAAXLDUDVM6CNVC76QWUT2VQKWFXOF5EL3O2EQA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.538823924 (190h17m30.497921172s) [noise=true , address=104.36.82.7:4912 , piece_id=HQSFCGME3FF7TUPOXD4GUJUQEF2Z5YWAZQGOAAZKILZBCGUBCW7Q, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.587643867 (190h17m30.546741114s) [noise=true , address=104.36.82.12:4149 , piece_id=QY7DO7YY7R3CG5P3P5UKU25SO6EYYIDB47EEN2MBWOKSWGR3NZ6Q, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.591865251 (190h17m30.550962501s) [noise=true , address=45.91.116.7:4011 , piece_id=L52OSPTBUDTGCSUMZIDL2EW54J7AIJYXITXC7QWCHYVN4NYEMT7Q, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.602422635 (190h17m30.561519889s) [noise=true , address=77.90.31.14:4663 , piece_id=TEGACRI7HO27B4ZI3QV62N7SIDSEQ4BE4ZQW77MKYV7O5GGKFJPQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.613558414 (190h17m30.572655661s) [noise=true , address=45.91.116.9:4261 , piece_id=4GL2AM3WJCUCTKABDOCKZYRJN32KPYGJPU3NTHOQNOSZ7FGGSWZA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.625333870 (190h17m30.584431127s) [noise=true , address=45.91.116.6:4974 , piece_id=ZEDZDURBN632VT57V5OQX5HNQE4EPEQPHW5PVRDO535EUDXPXTCQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.632179694 (190h17m30.591276939s) [noise=true , address=45.91.116.5:4297 , piece_id=5632ZHQWXBS66OW4CR5UHEH7DCJOYMRUQS3X37QF7FFOUX3JMJDA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.887245127 (190h17m30.846342377s) [noise=true , address=45.91.116.8:4048 , piece_id=J536X5BXF6OERIFMFNYLXR25QHKTWWJ7DPSNSIIP4QXMTYHAUWWA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:25.821185212 (190h17m47.780282468s) [noise=true , address=104.36.82.11:4115 , piece_id=ZZLZE25XOWBWJDDH7FEEYR74YPCVSBVI7CQD6GTG6CQMEFGHK2KA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:36.923306399 (190h17m58.882403632s) [noise=true , address=104.36.82.4:4804 , piece_id=3QYWY4AWHJX3U4WBOBQ2OKDBJ3KPC4YTWSKR3UJG2H7QKB3GLZYQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:37.351811885 (190h17m59.31090912s) [noise=true , address=104.36.82.2:4731 , piece_id=ZL4OSIWH7TC3OX4JUN6DFOYEKN7QCWJBIJAX4ROLPTDM3SGUDP2A, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:53300->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:53300->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53300->104.36.82.2:4731: use of closed network connection Aug 16 15:43:37.351930005 (190h17m59.311027244s) [noise=true , address=104.36.82.5:4839 , piece_id=P47CLM36SLM6E7UKL5YYRVUCTYMMGE63PHFN6QNQR4GVT325GR5A, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:54380->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:54380->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54380->104.36.82.5:4839: use of closed network connection Aug 16 15:43:37.352017308 (190h17m59.311114553s) [noise=true , address=77.90.31.18:4807 , piece_id=5HZXGFQK6ZMM6NW6SOT7UKVR5KWA2RKNPJYHG7WYBOK6KZ57JRIA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 139.59.144.22:41680->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:37.352103504 (190h17m59.311200756s) [noise=true , address=104.36.82.8:4971 , piece_id=WR6X7OIPS2522GSM6L6JG4OJRAOD5GMEETCJOFHEQCRKZZQROTCQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:43838->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:43838->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43838->104.36.82.8:4971: use of closed network connection Aug 16 15:43:37.352146090 (190h17m59.311243347s) [seg_pos={0 0}, upload=198084]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:43:37.460521948 (190h17m59.419619203s) [upload=198084]: Done uploading segments... Aug 16 15:43:37.460555789 (190h17m59.419653042s) [upload=198084]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.112354372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX...X..........................X............] duration: 38.022594213s FAIL (2025-08-16 15:43:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 38.022625185s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...........X....................] duration: 2.29859031s FAIL (2025-08-16 15:43:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 2.298635855s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)