Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.677744851s FAIL (2025-07-27 01:38:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 4.677795929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 871.015467ms FAIL (2025-07-27 01:38:33 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 871.072126ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.3018137s FAIL (2025-07-27 01:38:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.301921008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.421430954s FAIL (2025-07-27 01:38:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.421462658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 905.175444ms FAIL (2025-07-27 01:37:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 905.22068ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.286443922s FAIL (2025-07-27 01:37:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.28649017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX...X......................................................] duration: 2.442128941s FAIL (2025-07-27 01:37:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.442162023s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 5.924018752s FAIL (2025-07-27 01:37:42 UTC) orbiter warning: context deadline exceeded Duration: 5.924077472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.024508846s FAIL (2025-07-27 01:37:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.024549206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XX.X........................................................] duration: 4.065088279s FAIL (2025-07-27 01:37:10 UTC) orbiter warning: context deadline exceeded Duration: 4.065116503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X................................................X..........] duration: 1.092387773s FAIL (2025-07-27 01:37:04 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.092414471s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.266623589s FAIL (2025-07-27 01:36:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.266660844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.078129432s FAIL (2025-07-27 01:36:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.078162613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XX...X.....................................................] duration: 2.431238484s FAIL (2025-07-27 01:36:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.431263072s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.989177832s FAIL (2025-07-27 01:36:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.989206883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.278293444s FAIL (2025-07-27 01:36:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.278329108s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X................................................X.........] duration: 1.074634355s FAIL (2025-07-27 01:35:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.074668592s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.XX.X.......................................................] duration: 2.992984098s FAIL (2025-07-27 01:35:58 UTC) orbiter warning: context deadline exceeded Duration: 2.993006881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.261025784s FAIL (2025-07-27 01:35:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.261087787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.434988108s FAIL (2025-07-27 01:35:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.435030685s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.81341208s FAIL (2025-07-27 01:35:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.813496927s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XX...X....................................................] duration: 2.449785834s FAIL (2025-07-27 01:35:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.449821239s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.743621591s FAIL (2025-07-27 01:35:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.7436624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X................................................X........] duration: 877.492276ms FAIL (2025-07-27 01:35:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 877.520893ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XX...X...................................................] duration: 2.283545714s FAIL (2025-07-27 01:35:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.283579035s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.268026178s FAIL (2025-07-27 01:35:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.268057889s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.685350176s FAIL (2025-07-27 01:34:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.68542421s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..XX.X......................................................] duration: 3.019623168s FAIL (2025-07-27 01:34:47 UTC) orbiter warning: context deadline exceeded Duration: 3.019667091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XX...X..................................................] duration: 2.265852882s FAIL (2025-07-27 01:34:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.265900795s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.926933449s FAIL (2025-07-27 01:34:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.926964803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...XX.X.....................................................] duration: 3.03045655s FAIL (2025-07-27 01:34:13 UTC) orbiter warning: context deadline exceeded Duration: 3.030491694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X................................................X.......] duration: 900.039449ms FAIL (2025-07-27 01:34:08 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 900.067923ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.114274135s FAIL (2025-07-27 01:34:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.114336098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.376453193s FAIL (2025-07-27 01:34:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.376507358s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.783604294s FAIL (2025-07-27 01:33:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.78368921s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XX...X.................................................] duration: 2.287994037s FAIL (2025-07-27 01:33:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.288050495s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.737717422s FAIL (2025-07-27 01:33:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.7377473s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.427434122s FAIL (2025-07-27 01:33:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.427477008s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....XX.X....................................................] duration: 3.000684705s FAIL (2025-07-27 01:33:15 UTC) orbiter warning: context deadline exceeded Duration: 3.000716139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X................................................X......] duration: 859.57556ms FAIL (2025-07-27 01:32:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 859.60289ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.688290471s FAIL (2025-07-27 01:32:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 4.688325419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.283948954s FAIL (2025-07-27 01:32:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.283979228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X................................................X.....] duration: 866.920589ms FAIL (2025-07-27 01:32:35 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 866.953138ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XX...X................................................] duration: 2.321998253s FAIL (2025-07-27 01:32:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.322030804s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.259472784s FAIL (2025-07-27 01:32:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=58738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:52:18.276215606 (61h16m49.466514186s) [upload=58738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:52:18.276222851 (61h16m49.466521431s) [upload=58738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:52:18.424271736 (61h16m49.614570316s) [upload=58738]: Done uploading segments... Jul 27 00:52:18.424297914 (61h16m49.61459648s) [upload=58738]: Upload finished. err: <nil> Jul 27 00:52:43.091728703 (61h17m14.282027283s) [upload=58744]: Starting upload Jul 27 00:52:43.091805734 (61h17m14.282104306s) [upload=58744]: Done starting upload Jul 27 00:52:43.091905456 (61h17m14.282204027s) [upload=58744]: Uploading segments... Jul 27 00:52:43.092150622 (61h17m14.282449192s) [upload=58744]: Got next segment. Inline: false Jul 27 00:52:43.092211374 (61h17m14.282509963s) [upload=58744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:52:43.092225110 (61h17m14.28252378s) [upload=58744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:52:43.204376996 (61h17m14.394675591s) [seg_pos={0 0}, upload=58744]: Begin upload segment... Jul 27 00:52:43.204437409 (61h17m14.394735987s) [seg_pos={0 0}, upload=58744]: Done begin upload segment. Jul 27 00:52:43.204471719 (61h17m14.394770302s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204681268 (61h17m14.394979862s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.204856017 (61h17m14.395154603s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205153431 (61h17m14.395452015s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205396626 (61h17m14.395695237s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205542855 (61h17m14.395841439s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.205838313 (61h17m14.396136895s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206021213 (61h17m14.396319804s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206168131 (61h17m14.396466713s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206472677 (61h17m14.396771268s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206494443 (61h17m14.396793032s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206818458 (61h17m14.39711704s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.206977782 (61h17m14.397276374s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.207426568 (61h17m14.397725158s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208620144 (61h17m14.398918729s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.208943500 (61h17m14.39924208s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Uploading piece... Jul 27 00:52:43.821906006 (61h17m15.012204577s) [upload=58744]: Next returned nil segment Jul 27 00:52:43.821963634 (61h17m15.012262223s) [upload=58744]: Waiting for error group managing segments... Jul 27 00:52:43.956659680 (61h17m15.146958272s) [noise=true , address=77.90.31.14:4663 , piece_id=PQRFWIL5K2VF24KBUC56WQ6AFX66DSRFJOUPJ7V3NX3LUFZPGBYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.958163784 (61h17m15.148462357s) [noise=true , address=45.91.116.10:4081 , piece_id=X7GGVUZQUWJ5F6HCDEOCHYQHG7PAWYMTP3MTSGTZ4EJ3K23N2TAQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.959512721 (61h17m15.149811303s) [noise=true , address=77.90.31.13:4627 , piece_id=7WIU7VAQORZ6X6SHVH3MNJKSR7DNR3O5BPU2VEK3ICOOQYKGRKUQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.960716709 (61h17m15.151015298s) [noise=true , address=45.91.116.9:4262 , piece_id=3IHFKQHAHO45IB7LQBEA4UOSNSULQ53GN6VAISDW76PY4NZWRWOA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.985765200 (61h17m15.176063792s) [noise=true , address=45.91.116.7:4010 , piece_id=7FCGTF4VSJ63JBQIGWDFIFL6POQ2KD67Z6V54GMG24QDHOKRYQLA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999062624 (61h17m15.189361197s) [noise=true , address=45.91.116.14:4189 , piece_id=KCBOJ32STCFZNAUTW5X5DPF27PM6BSFRNXNTYBNDHYNI6QNNQCHA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:43.999550987 (61h17m15.189849573s) [noise=true , address=45.91.116.5:4299 , piece_id=2R2DEFJZ7DXD33INVQQ6AHG2A5NGUQF4CLWLAIZDDX4PLLFMPA6A, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.003313228 (61h17m15.193611807s) [noise=true , address=45.91.116.4:4938 , piece_id=UAKRTQEM4RR5H3J5PRMG7WHARNOGQGXXYLWT3KU36NXVLGIJXONQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.012056927 (61h17m15.202355505s) [noise=true , address=45.91.116.8:4047 , piece_id=DSXUADQYOKBVDHYSM4NYE4KVR7E7OFIRL36QHKZLR6IF35CIRHZA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.040860082 (61h17m15.23115866s) [noise=true , address=45.91.116.12:4225 , piece_id=ZBCGG6RZ6BWJGNVQHEQWCUKT67W75ZCLOYNNVCKZ5KQLTCXKXZYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:44.041016157 (61h17m15.231314738s) [noise=true , address=45.91.116.9:4263 , piece_id=J2V7HHZ6T57JO67MELCHQJDAZSNRZ4WMTD4HYGATQJ6QJ6TQYRYQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:52:45.975687665 (61h17m17.16598623s) [noise=true , address=77.90.31.12:4592 , piece_id=SYINBLX4C5YEFFHC77NEFY4D4OJ7KTYEWADPW2R7PAOBIFZT24EA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: <nil> Jul 27 00:53:12.818505138 (61h17m44.008803722s) [noise=true , address=104.36.82.2:4731 , piece_id=6XI5354ZYMCYBAUADWF22N6JQXOOU6HMZST2RATU5UCH34C35XKA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:53:12.818640577 (61h17m44.008939156s) [noise=true , address=104.36.82.8:4971 , piece_id=RAU3Z4A25DEQ7NULSKBUHRD54ITWROHZFF5OFTAXN4MFI4J4RGYA, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57502->104.36.82.8:4971: use of closed network connection Jul 27 00:53:12.818709163 (61h17m44.009007743s) [noise=true , address=104.36.82.7:4910 , piece_id=N525YQMBOOJIPENV5DMHQ5HXC4ZIE5FKEBNHGBSGIIPRH6TIHEDQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35568->104.36.82.7:4910: use of closed network connection Jul 27 00:53:12.818867239 (61h17m44.009165835s) [noise=true , address=104.36.82.14:4221 , piece_id=3PVW4ILMVDQQJZI4EYFKGYO5F5IZWS37TXRURX6MBWQGKBQA2OVQ, seg_pos={0 0}, upload=58744]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35422->104.36.82.14:4221: use of closed network connection Jul 27 00:53:12.818911112 (61h17m44.009209701s) [seg_pos={0 0}, upload=58744]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:53:12.931161973 (61h17m44.121460554s) [upload=58744]: Done uploading segments... Jul 27 00:53:12.931207441 (61h17m44.12150602s) [upload=58744]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.259513985s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.667015022s FAIL (2025-07-27 01:32:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.667042479s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....XX.X...................................................] duration: 2.742384171s FAIL (2025-07-27 01:32:10 UTC) orbiter warning: context deadline exceeded Duration: 2.742436583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.047056248s FAIL (2025-07-27 01:32:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== h19m34.727341673s) [upload=58775]: Upload finished. err: <nil> Jul 27 00:55:33.730682950 (61h20m4.920981518s) [upload=58792]: Starting upload Jul 27 00:55:33.730756046 (61h20m4.921054625s) [upload=58792]: Done starting upload Jul 27 00:55:33.730987051 (61h20m4.921285623s) [upload=58792]: Uploading segments... Jul 27 00:55:33.731109876 (61h20m4.921408446s) [upload=58792]: Got next segment. Inline: false Jul 27 00:55:33.731173971 (61h20m4.921472553s) [upload=58792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:55:33.731180572 (61h20m4.921479144s) [upload=58792]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:55:33.921863592 (61h20m5.112162174s) [seg_pos={0 0}, upload=58792]: Begin upload segment... Jul 27 00:55:33.922005068 (61h20m5.112303657s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.922603942 (61h20m5.112902528s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923003765 (61h20m5.11330235s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923323491 (61h20m5.11362206s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.923636195 (61h20m5.113934765s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924087678 (61h20m5.114386262s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.924784214 (61h20m5.115082795s) [seg_pos={0 0}, upload=58792]: Done begin upload segment. Jul 27 00:55:33.924811678 (61h20m5.115110263s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.926675303 (61h20m5.116973888s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927244451 (61h20m5.11754304s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.927711403 (61h20m5.118009989s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928096166 (61h20m5.118394737s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928482779 (61h20m5.118781368s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.928896321 (61h20m5.119194901s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929144054 (61h20m5.11944264s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929807549 (61h20m5.120106133s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.929957205 (61h20m5.120255774s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.930650688 (61h20m5.120949257s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:33.931470071 (61h20m5.121768651s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Uploading piece... Jul 27 00:55:34.467360827 (61h20m5.657659447s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IMVAFRZMCRJDOIVAOM3R7BSLZ7MJ5F3IJC7JGQKD23A55RP3GQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:34.482323354 (61h20m5.672621923s) [upload=58792]: Next returned nil segment Jul 27 00:55:34.482820843 (61h20m5.673119422s) [upload=58792]: Waiting for error group managing segments... Jul 27 00:55:35.128025950 (61h20m6.318324528s) [noise=true , address=79.127.201.214:4313 , piece_id=LQUMZZI4VMKY4MJYW3PMMNBGE5EYGYR3Y6VTGAGF4FCXXNJ5CARA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.365643389 (61h20m6.555942046s) [noise=true , address=79.127.205.234:4604 , piece_id=MHQDRHZ7HI5LCUP2RQXBEEG5WUEJUJYDRETWQMMOEPHMRX5L4ZEQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.406048796 (61h20m6.596347365s) [noise=true , address=79.127.205.246:4266 , piece_id=JBDM4WMTZUWCZWVD66H2UWTOYLP4DGHZXYLCJE4QSPOMTFSONH2Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.451230307 (61h20m6.64152888s) [noise=true , address=79.127.205.233:4653 , piece_id=6JRFERZ2BXTANFF3CZ55GG324JSKRNNPLWHOCYUAS5VANWQ2MD7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:55:35.763914117 (61h20m6.954212702s) [noise=true , address=64.49.13.140:4508 , piece_id=3244R75BNZDCTNJGBJOZLCJT5MMRTIKCIU43CW62HDGEG7GV5IHQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:55:59.212577487 (61h20m30.402876077s) [noise=true , address=104.36.82.4:4826 , piece_id=JKFTRE7W526IOGVPAYMUQ3744SHH6U7FHUR3JJ4YFSGB23NU6UPA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:00.113616178 (61h20m31.303914748s) [noise=true , address=104.36.82.54:4164 , piece_id=ZR6VHRGUFCUEC24U2TIZSV7F7G5ZQANMQUGSPYAUBCBSRN42YAOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:03.630253934 (61h20m34.820552506s) [noise=true , address=104.36.82.53:4122 , piece_id=RXHVABALFKGKO7TAUYA4MHZALT62XOROSLVAK5W3GIHRX5DXDC6A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:04.298647855 (61h20m35.48894642s) [noise=true , address=104.36.82.55:4228 , piece_id=E7EGSPWNW7YOJETI34L4K3WA6F67SBAO4CTS4O2PD5S5RDPLS7IA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:17.871789142 (61h20m49.062087716s) [noise=true , address=104.36.82.46:4483 , piece_id=U6H2GZ56A3XWLLWCRPKVUXRM7BYIIQ6W5VD64MVP5AQI5ZMANJOQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:21.322592849 (61h20m52.512891421s) [noise=true , address=77.90.31.16:4760 , piece_id=Z2GXZZXLCXGJADUWVB4RFKCDEIKBH5JYU37SEOMCBC7MXNE43W7A, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:27.578141431 (61h20m58.768440015s) [noise=true , address=77.90.31.21:4908 , piece_id=DV25ZEGS6LDHGNN663GMXQXQR4FZQLEOFSZIPGIIFVCBEGYXO77Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:32.537706298 (61h21m3.728004872s) [noise=true , address=104.36.82.34:4566 , piece_id=XQWJ7NVIUBQQAQ6VQYQ3ZOKRFRHGKGZIWVFUABDL7LJHAYQQKM7Q, seg_pos={0 0}, upload=58792]: Done uploading piece. err: <nil> Jul 27 00:56:33.459124932 (61h21m4.649423511s) [noise=true , address=104.36.82.41:4766 , piece_id=QCCIBWJA3A6W7NA6TPEDO2AVD3JWUCL2IOSYOOBCJIN4A4WVPYAA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459219503 (61h21m4.649518072s) [noise=true , address=104.36.82.42:4817 , piece_id=24YVTYBWQGRSPKJRXO7YOCCK4RVQMXDKWHLLT6HS5OFBWIOP3WBA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459335276 (61h21m4.649633846s) [noise=true , address=104.36.82.47:4519 , piece_id=6IZ4QV2IYGBEAYCJX35EAWJAX5WDJLCKNNODFPPVQB5MEXNFVYOA, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459470602 (61h21m4.649769181s) [noise=true , address=104.36.82.9:4015 , piece_id=SUTEQSAD4QVBOPA662VNEVTKRONFFAU5FGMCNXNYDBKC4QKDDPLQ, seg_pos={0 0}, upload=58792]: Done uploading piece. err: ecclient: upload failed (node:1KXaAs7ZW21xGPAZRVUxGMBgH78eejQ2kxEm5CQR5ZdQvzZd6o, address:104.36.82.9:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:56:33.459495866 (61h21m4.649794446s) [seg_pos={0 0}, upload=58792]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:56:33.566076266 (61h21m4.756374846s) [upload=58792]: Done uploading segments... Jul 27 00:56:33.566125923 (61h21m4.756424503s) [upload=58792]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.04710961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X................................................X....] duration: 860.375908ms FAIL (2025-07-27 01:31:36 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 860.415379ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XX...X...............................................] duration: 2.401349231s FAIL (2025-07-27 01:31:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.401378257s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)