Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 832.335245ms FAIL (2025-07-26 22:56:08 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 832.367843ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.............................X............] duration: 2.332327024s FAIL (2025-07-26 22:56:05 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.332375024s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [...X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X] duration: 1m0.108876721s FAIL (2025-07-26 22:55:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 3659698 (59h19m24.843958284s) [upload=56894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:54:53.895339399 (59h19m25.085637977s) [seg_pos={0 0}, upload=56894]: Begin upload segment... Jul 26 22:54:53.895397422 (59h19m25.085696005s) [seg_pos={0 0}, upload=56894]: Done begin upload segment. Jul 26 22:54:53.895422888 (59h19m25.08572147s) [noise=true , address=64.49.13.155:4918 , piece_id=X3J27AQVB66OB7TC3TAVIVXUXP2H36ADLV6EI3OOPKAUM64W5YGA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.895491522 (59h19m25.085790101s) [noise=true , address=45.91.116.9:4273 , piece_id=N5J6MYR4NEG5A4EXFEPEVNOI4EIJRY7QTCXBU5JMFZKEPVXJVTKA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.895957268 (59h19m25.08625585s) [noise=true , address=77.90.31.17:4784 , piece_id=25GWTVWEGSA7344OLNCYSPKVCVG3D6WWK3AUAT3TBLHX22XY37ZQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.896262941 (59h19m25.08656152s) [noise=true , address=64.49.13.26:4951 , piece_id=PQQBGTEP27MXZ3YY7L5GQJL6FX56GJVHBCI27ZB6OIHKMJRSIGLA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.896811576 (59h19m25.087110159s) [noise=true , address=64.49.13.140:4508 , piece_id=DGUUC5ABLL7TBJEGUBLMIJFMWJQ76E3Q4KHTOXF3CJJI7UKDPV5Q, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897286536 (59h19m25.087585115s) [noise=true , address=79.127.219.38:4006 , piece_id=T5L2FDRKRWYTQGBNF7UAR774AEA47GI4OXX7KBNHMCRNNTLNB4XA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897379252 (59h19m25.087677838s) [noise=true , address=45.91.116.14:4194 , piece_id=FOWBWZJQ2QN7KD3AK5UKTW2CWKARYW3HADAL46CO3IAXTSKIZSYQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897758399 (59h19m25.088056978s) [noise=true , address=64.49.13.140:4514 , piece_id=YRPYCOA7MALDIETIMT3ZCWNRDBQ34ZAG6J4E4X2REVUD7G43VXXQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897766108 (59h19m25.088064693s) [noise=true , address=79.127.201.215:4352 , piece_id=4JXTKYGEZKITND4KEEXOJKGFCNBFH5M7O55A65RTIVYRDAMNJZHA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898175302 (59h19m25.088473884s) [noise=true , address=45.91.116.14:4206 , piece_id=FXPC4WAH2MYZXAE2JW47WFMCJWA7CSGYJZ3FTAETJHOI35Y7T3VA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898282568 (59h19m25.08858115s) [noise=true , address=77.90.31.15:4703 , piece_id=77BNC2ZFGXN3X67AW7JFH2555EJJB4UCWADXZUE6AL7UVZ2JGVUA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898532129 (59h19m25.088830715s) [noise=true , address=79.127.205.233:4629 , piece_id=37HF3OWKWH2FHKPRHWWJ5N7YNCIMWWPCI7QBZV4ZBOBC62X2EUHA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898745859 (59h19m25.089044441s) [noise=true , address=79.127.220.101:4048 , piece_id=4KAJHO6NSQ7YGKYK3OMJU5DHN2ZDUIXRLBWHMFIDFHU42XTV5IEQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898843412 (59h19m25.089141996s) [noise=true , address=45.91.116.11:4121 , piece_id=II7ELY7KO5IYEQKPPU4GXLWUAHWEGZT37HD3MKSK6SH2336GQ64Q, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899145378 (59h19m25.089443962s) [noise=true , address=79.127.219.51:4212 , piece_id=7DR6RDCRSWLGSL4LRCJHFA5X5ZXBKYRRE4XUC7R6W3HBGDECXYRQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899291753 (59h19m25.089590332s) [noise=true , address=77.90.31.19:4867 , piece_id=WWBDA2NJGZ4IY7O2EFKVVIMUG6IDQ6F7AB2ILIHEU23BAVOSVK4A, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899501827 (59h19m25.089800411s) [noise=true , address=79.127.205.246:4281 , piece_id=MBPFA3HHSDTWC3B6GKWNTW4HSUQNJXU6HQNBIVOC7GONWQAEUHEA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899699595 (59h19m25.089998174s) [noise=true , address=156.146.43.225:4449 , piece_id=KPISBKWXFORIUITPSSXGILEYCPXLMKSXQT6XBSTS7FVSXXUU7OYQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:54.417137519 (59h19m25.607436113s) [noise=true , address=79.127.220.101:4048 , piece_id=4KAJHO6NSQ7YGKYK3OMJU5DHN2ZDUIXRLBWHMFIDFHU42XTV5IEQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:54.454141381 (59h19m25.644439961s) [noise=true , address=156.146.43.225:4449 , piece_id=KPISBKWXFORIUITPSSXGILEYCPXLMKSXQT6XBSTS7FVSXXUU7OYQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:54.457574476 (59h19m25.647873113s) [upload=56894]: Next returned nil segment Jul 26 22:54:54.457593123 (59h19m25.647891701s) [upload=56894]: Waiting for error group managing segments... Jul 26 22:54:54.615981623 (59h19m25.806280202s) [noise=true , address=64.49.13.155:4918 , piece_id=X3J27AQVB66OB7TC3TAVIVXUXP2H36ADLV6EI3OOPKAUM64W5YGA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK, address:64.49.13.155:4918): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:54.620288789 (59h19m25.810587358s) [noise=true , address=45.91.116.14:4206 , piece_id=FXPC4WAH2MYZXAE2JW47WFMCJWA7CSGYJZ3FTAETJHOI35Y7T3VA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:55.155310877 (59h19m26.34560946s) [noise=true , address=64.49.13.26:4951 , piece_id=PQQBGTEP27MXZ3YY7L5GQJL6FX56GJVHBCI27ZB6OIHKMJRSIGLA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:55.194851814 (59h19m26.385150395s) [noise=true , address=79.127.205.246:4281 , piece_id=MBPFA3HHSDTWC3B6GKWNTW4HSUQNJXU6HQNBIVOC7GONWQAEUHEA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:55.214815830 (59h19m26.4051144s) [noise=true , address=79.127.205.233:4629 , piece_id=37HF3OWKWH2FHKPRHWWJ5N7YNCIMWWPCI7QBZV4ZBOBC62X2EUHA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:55.792814728 (59h19m26.983113317s) [noise=true , address=45.91.116.9:4273 , piece_id=N5J6MYR4NEG5A4EXFEPEVNOI4EIJRY7QTCXBU5JMFZKEPVXJVTKA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:56.233265946 (59h19m27.423564525s) [noise=true , address=79.127.219.38:4006 , piece_id=T5L2FDRKRWYTQGBNF7UAR774AEA47GI4OXX7KBNHMCRNNTLNB4XA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:56.269645596 (59h19m27.459944168s) [noise=true , address=77.90.31.15:4703 , piece_id=77BNC2ZFGXN3X67AW7JFH2555EJJB4UCWADXZUE6AL7UVZ2JGVUA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:57.940034816 (59h19m29.130333398s) [noise=true , address=79.127.219.51:4212 , piece_id=7DR6RDCRSWLGSL4LRCJHFA5X5ZXBKYRRE4XUC7R6W3HBGDECXYRQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:58.788209768 (59h19m29.978508336s) [noise=true , address=64.49.13.140:4514 , piece_id=YRPYCOA7MALDIETIMT3ZCWNRDBQ34ZAG6J4E4X2REVUD7G43VXXQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:59.398611894 (59h19m30.588910459s) [noise=true , address=45.91.116.11:4121 , piece_id=II7ELY7KO5IYEQKPPU4GXLWUAHWEGZT37HD3MKSK6SH2336GQ64Q, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:55:05.791624622 (59h19m36.981923195s) [noise=true , address=64.49.13.140:4508 , piece_id=DGUUC5ABLL7TBJEGUBLMIJFMWJQ76E3Q4KHTOXF3CJJI7UKDPV5Q, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:55:43.211019053 (59h20m14.401317622s) [noise=true , address=79.127.201.215:4352 , piece_id=4JXTKYGEZKITND4KEEXOJKGFCNBFH5M7O55A65RTIVYRDAMNJZHA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:55:53.372353686 (59h20m24.562652275s) [noise=true , address=77.90.31.19:4867 , piece_id=WWBDA2NJGZ4IY7O2EFKVVIMUG6IDQ6F7AB2ILIHEU23BAVOSVK4A, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:55:53.372645226 (59h20m24.562943805s) [noise=true , address=77.90.31.17:4784 , piece_id=25GWTVWEGSA7344OLNCYSPKVCVG3D6WWK3AUAT3TBLHX22XY37ZQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: write tcp 139.59.144.22:33738->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:55:53.372722799 (59h20m24.563021379s) [noise=true , address=45.91.116.14:4194 , piece_id=FOWBWZJQ2QN7KD3AK5UKTW2CWKARYW3HADAL46CO3IAXTSKIZSYQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: write tcp 139.59.144.22:50024->45.91.116.14:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:55:53.372757166 (59h20m24.563055752s) [seg_pos={0 0}, upload=56894]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:55:53.478852984 (59h20m24.669151559s) [upload=56894]: Done uploading segments... Jul 26 22:55:53.478900783 (59h20m24.669199364s) [upload=56894]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 1m0.108899087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.313928065s FAIL (2025-07-26 22:55:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.313963621s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [............X.......X..X...X.XX...X..X.XX......X.X.......X..] duration: 14.615495486s FAIL (2025-07-26 22:55:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.615531477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.357734283s FAIL (2025-07-26 22:55:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.357765284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.............................X...........] duration: 2.295878925s FAIL (2025-07-26 22:55:17 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.295905731s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 928.540831ms FAIL (2025-07-26 22:55:09 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 928.69724ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [....X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.] duration: 52.770023427s FAIL (2025-07-26 22:54:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 2:53:00.673887210 (59h17m31.864185801s) [noise=true , address=77.90.31.10:4528 , piece_id=T6XAYVQOZR2CP7YSNREVYRYHFJXH5LPKC43RNVEA5P7HLBXOSSYQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.674394447 (59h17m31.864693037s) [noise=true , address=79.127.220.97:4361 , piece_id=UAC4MKLGHCW2T4CAZ44XPU6HFNV46XZ2VEHDO7RCGGUPUH5PAMLQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.674796432 (59h17m31.865095021s) [noise=true , address=77.90.31.11:4564 , piece_id=VPM3YDZOHEQ2K4VGDFTSP2X5VB7GMSUHYPIC6NGNNZYHNCPZDX4Q, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.675170480 (59h17m31.86546907s) [noise=true , address=77.90.31.18:4824 , piece_id=B6D7F7CUEBVMEPZ5SANZMDVT4QVJGC4RJSE2PVTYT7QOOGZEPLQQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.675558348 (59h17m31.865856934s) [noise=true , address=79.127.219.51:4212 , piece_id=XQS3FKUB2SVFS3FJMPARQEL2C5GIBOZJUC7ZKCAIMP5MLIEK6CCA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.675908319 (59h17m31.866206909s) [noise=true , address=77.90.31.17:4791 , piece_id=JCQDLV3LFITU3R3FVKCFF2Y7JRDIAO674NUON45SWN2EH4BTXGMQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.678086332 (59h17m31.868384987s) [seg_pos={0 0}, upload=56864]: Done begin upload segment. Jul 26 22:53:00.678167461 (59h17m31.868466177s) [noise=true , address=79.127.205.234:4620 , piece_id=BETZASUWANWUZJMWNWBH4Q6F2FEEF3TVYPEFI7LJSWIU4TANWUEA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.678555612 (59h17m31.868854178s) [noise=true , address=45.91.116.9:4267 , piece_id=DBRX7SPXRIPZWF3R336D3C4V5Y3RXTDOK7D5TRGELNCIGN2GJOGA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.678933808 (59h17m31.86923239s) [noise=true , address=64.49.13.140:4515 , piece_id=LFNDH2JP7BHTNQB4RFAJIGVI76F5KKZEWSVSCSXMRQP7WPWD5POA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.679283622 (59h17m31.869582211s) [noise=true , address=45.91.116.14:4222 , piece_id=RCIFS6AAO6F24MDP5L3R4GLLP2ZPXV7KDJ6FIVVFOMJQRXVJ3VJA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.679588899 (59h17m31.869887481s) [noise=true , address=79.127.219.51:4210 , piece_id=FPTLF2X6JMP7AKQ3IRR2ZGFGBR2N6CV4HG4CQKAQJESORRTY43PQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.679685649 (59h17m31.869984237s) [noise=true , address=79.127.219.38:4015 , piece_id=CEN5SHDZPFIR7ET3IQMZJ7XXMYHK452WT4NRPT6EYV5AOQ3JQIBQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.680129695 (59h17m31.870428281s) [noise=true , address=45.91.116.8:4065 , piece_id=JJXC676BBRD3A34NQVHHRLQYACEURAQBGQIH4STVE3KRJSAAR34Q, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.680547824 (59h17m31.870846416s) [noise=true , address=45.91.116.11:4124 , piece_id=34S6H73HSOROLG4M3ZV3RR2TODJVF336O33VJDBRGGXSXXAJ7BHQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.680869781 (59h17m31.871168367s) [noise=true , address=79.127.219.47:4048 , piece_id=7QN4AGPILTMSZN7G5CHXPIOVUYBNFGLECPPIJCIKRLD7DSYLKBOQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.681186032 (59h17m31.87148462s) [noise=true , address=156.146.43.225:4449 , piece_id=2BLCW4PNLHMCTBKO3DDY2UWWGQ57R4GQYR5CJOJ4F3TKJQRHOOHA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.681427694 (59h17m31.871726259s) [noise=true , address=64.49.13.138:4482 , piece_id=NDC2CRBXQMAWY3S3GMR6IE6YZIJ6E3S6J4NR4ZTZG3YUOCWF3FUQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.681734231 (59h17m31.872032817s) [noise=true , address=77.90.31.16:4767 , piece_id=24TFO6NBPXF2B2734DTA4V3SISXM6IBBUVGL7IXHPZ5PBQJZJQRQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:01.088438432 (59h17m32.278737016s) [noise=true , address=79.127.219.51:4212 , piece_id=XQS3FKUB2SVFS3FJMPARQEL2C5GIBOZJUC7ZKCAIMP5MLIEK6CCA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:01.190866356 (59h17m32.381164934s) [noise=true , address=79.127.219.51:4210 , piece_id=FPTLF2X6JMP7AKQ3IRR2ZGFGBR2N6CV4HG4CQKAQJESORRTY43PQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:01.237479027 (59h17m32.427777598s) [upload=56864]: Next returned nil segment Jul 26 22:53:01.237509085 (59h17m32.42780767s) [upload=56864]: Waiting for error group managing segments... Jul 26 22:53:01.388060980 (59h17m32.578359565s) [noise=true , address=79.127.219.38:4015 , piece_id=CEN5SHDZPFIR7ET3IQMZJ7XXMYHK452WT4NRPT6EYV5AOQ3JQIBQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:01.477019570 (59h17m32.667318385s) [noise=true , address=77.90.31.11:4564 , piece_id=VPM3YDZOHEQ2K4VGDFTSP2X5VB7GMSUHYPIC6NGNNZYHNCPZDX4Q, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:01.570426403 (59h17m32.760724994s) [noise=true , address=79.127.220.97:4361 , piece_id=UAC4MKLGHCW2T4CAZ44XPU6HFNV46XZ2VEHDO7RCGGUPUH5PAMLQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:02.443045146 (59h17m33.633343742s) [noise=true , address=79.127.205.234:4620 , piece_id=BETZASUWANWUZJMWNWBH4Q6F2FEEF3TVYPEFI7LJSWIU4TANWUEA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:02.553994078 (59h17m33.744292657s) [noise=true , address=45.91.116.8:4065 , piece_id=JJXC676BBRD3A34NQVHHRLQYACEURAQBGQIH4STVE3KRJSAAR34Q, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:03.797194512 (59h17m34.987493083s) [noise=true , address=64.49.13.140:4515 , piece_id=LFNDH2JP7BHTNQB4RFAJIGVI76F5KKZEWSVSCSXMRQP7WPWD5POA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:04.762522644 (59h17m35.952821225s) [noise=true , address=64.49.13.138:4482 , piece_id=NDC2CRBXQMAWY3S3GMR6IE6YZIJ6E3S6J4NR4ZTZG3YUOCWF3FUQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:05.113474688 (59h17m36.303773258s) [noise=true , address=79.127.219.47:4048 , piece_id=7QN4AGPILTMSZN7G5CHXPIOVUYBNFGLECPPIJCIKRLD7DSYLKBOQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:05.221436882 (59h17m36.411735455s) [noise=true , address=156.146.43.225:4449 , piece_id=2BLCW4PNLHMCTBKO3DDY2UWWGQ57R4GQYR5CJOJ4F3TKJQRHOOHA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:06.110375991 (59h17m37.300674561s) [noise=true , address=45.91.116.9:4267 , piece_id=DBRX7SPXRIPZWF3R336D3C4V5Y3RXTDOK7D5TRGELNCIGN2GJOGA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:07.366873835 (59h17m38.557172407s) [noise=true , address=77.90.31.16:4767 , piece_id=24TFO6NBPXF2B2734DTA4V3SISXM6IBBUVGL7IXHPZ5PBQJZJQRQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:09.150232925 (59h17m40.340531505s) [noise=true , address=45.91.116.11:4124 , piece_id=34S6H73HSOROLG4M3ZV3RR2TODJVF336O33VJDBRGGXSXXAJ7BHQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:09.300314202 (59h17m40.490612781s) [noise=true , address=45.91.116.14:4222 , piece_id=RCIFS6AAO6F24MDP5L3R4GLLP2ZPXV7KDJ6FIVVFOMJQRXVJ3VJA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:00.192255385 (59h18m31.38255414s) [noise=true , address=77.90.31.18:4824 , piece_id=B6D7F7CUEBVMEPZ5SANZMDVT4QVJGC4RJSE2PVTYT7QOOGZEPLQQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 139.59.144.22:46048->77.90.31.18:4824: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:54:00.192355963 (59h18m31.382654543s) [noise=true , address=77.90.31.10:4528 , piece_id=T6XAYVQOZR2CP7YSNREVYRYHFJXH5LPKC43RNVEA5P7HLBXOSSYQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:42790->77.90.31.10:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:54:00.192668569 (59h18m31.382967161s) [noise=true , address=77.90.31.17:4791 , piece_id=JCQDLV3LFITU3R3FVKCFF2Y7JRDIAO674NUON45SWN2EH4BTXGMQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: read tcp 139.59.144.22:50080->77.90.31.17:4791: use of closed network connection; context deadline exceeded Jul 26 22:54:00.192755106 (59h18m31.383053703s) [seg_pos={0 0}, upload=56864]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:54:00.300405600 (59h18m31.490704167s) [upload=56864]: Done uploading segments... Jul 26 22:54:00.300432148 (59h18m31.490730727s) [upload=56864]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 52.770050122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 811.031811ms FAIL (2025-07-26 22:54:51 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 811.093103ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.............X.......X..X...X.XX...X..X.XX......X.X.......X.] duration: 12.019047074s FAIL (2025-07-26 22:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.019066685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.............................X..........] duration: 2.244691171s FAIL (2025-07-26 22:54:42 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.244722006s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..............X.......X..X...X.XX...X..X.XX......X.X.......X] duration: 1m0.001254864s FAIL (2025-07-26 22:54:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001277893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 849.586312ms FAIL (2025-07-26 22:54:30 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 849.612456ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.205425712s FAIL (2025-07-26 22:54:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 6.205455279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [.....X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX] duration: 1m0.110632549s FAIL (2025-07-26 22:54:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 2:53:00.673887210 (59h17m31.864185801s) [noise=true , address=77.90.31.10:4528 , piece_id=T6XAYVQOZR2CP7YSNREVYRYHFJXH5LPKC43RNVEA5P7HLBXOSSYQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.674394447 (59h17m31.864693037s) [noise=true , address=79.127.220.97:4361 , piece_id=UAC4MKLGHCW2T4CAZ44XPU6HFNV46XZ2VEHDO7RCGGUPUH5PAMLQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.674796432 (59h17m31.865095021s) [noise=true , address=77.90.31.11:4564 , piece_id=VPM3YDZOHEQ2K4VGDFTSP2X5VB7GMSUHYPIC6NGNNZYHNCPZDX4Q, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.675170480 (59h17m31.86546907s) [noise=true , address=77.90.31.18:4824 , piece_id=B6D7F7CUEBVMEPZ5SANZMDVT4QVJGC4RJSE2PVTYT7QOOGZEPLQQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.675558348 (59h17m31.865856934s) [noise=true , address=79.127.219.51:4212 , piece_id=XQS3FKUB2SVFS3FJMPARQEL2C5GIBOZJUC7ZKCAIMP5MLIEK6CCA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.675908319 (59h17m31.866206909s) [noise=true , address=77.90.31.17:4791 , piece_id=JCQDLV3LFITU3R3FVKCFF2Y7JRDIAO674NUON45SWN2EH4BTXGMQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.678086332 (59h17m31.868384987s) [seg_pos={0 0}, upload=56864]: Done begin upload segment. Jul 26 22:53:00.678167461 (59h17m31.868466177s) [noise=true , address=79.127.205.234:4620 , piece_id=BETZASUWANWUZJMWNWBH4Q6F2FEEF3TVYPEFI7LJSWIU4TANWUEA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.678555612 (59h17m31.868854178s) [noise=true , address=45.91.116.9:4267 , piece_id=DBRX7SPXRIPZWF3R336D3C4V5Y3RXTDOK7D5TRGELNCIGN2GJOGA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.678933808 (59h17m31.86923239s) [noise=true , address=64.49.13.140:4515 , piece_id=LFNDH2JP7BHTNQB4RFAJIGVI76F5KKZEWSVSCSXMRQP7WPWD5POA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.679283622 (59h17m31.869582211s) [noise=true , address=45.91.116.14:4222 , piece_id=RCIFS6AAO6F24MDP5L3R4GLLP2ZPXV7KDJ6FIVVFOMJQRXVJ3VJA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.679588899 (59h17m31.869887481s) [noise=true , address=79.127.219.51:4210 , piece_id=FPTLF2X6JMP7AKQ3IRR2ZGFGBR2N6CV4HG4CQKAQJESORRTY43PQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.679685649 (59h17m31.869984237s) [noise=true , address=79.127.219.38:4015 , piece_id=CEN5SHDZPFIR7ET3IQMZJ7XXMYHK452WT4NRPT6EYV5AOQ3JQIBQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.680129695 (59h17m31.870428281s) [noise=true , address=45.91.116.8:4065 , piece_id=JJXC676BBRD3A34NQVHHRLQYACEURAQBGQIH4STVE3KRJSAAR34Q, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.680547824 (59h17m31.870846416s) [noise=true , address=45.91.116.11:4124 , piece_id=34S6H73HSOROLG4M3ZV3RR2TODJVF336O33VJDBRGGXSXXAJ7BHQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.680869781 (59h17m31.871168367s) [noise=true , address=79.127.219.47:4048 , piece_id=7QN4AGPILTMSZN7G5CHXPIOVUYBNFGLECPPIJCIKRLD7DSYLKBOQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.681186032 (59h17m31.87148462s) [noise=true , address=156.146.43.225:4449 , piece_id=2BLCW4PNLHMCTBKO3DDY2UWWGQ57R4GQYR5CJOJ4F3TKJQRHOOHA, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.681427694 (59h17m31.871726259s) [noise=true , address=64.49.13.138:4482 , piece_id=NDC2CRBXQMAWY3S3GMR6IE6YZIJ6E3S6J4NR4ZTZG3YUOCWF3FUQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:00.681734231 (59h17m31.872032817s) [noise=true , address=77.90.31.16:4767 , piece_id=24TFO6NBPXF2B2734DTA4V3SISXM6IBBUVGL7IXHPZ5PBQJZJQRQ, seg_pos={0 0}, upload=56864]: Uploading piece... Jul 26 22:53:01.088438432 (59h17m32.278737016s) [noise=true , address=79.127.219.51:4212 , piece_id=XQS3FKUB2SVFS3FJMPARQEL2C5GIBOZJUC7ZKCAIMP5MLIEK6CCA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:01.190866356 (59h17m32.381164934s) [noise=true , address=79.127.219.51:4210 , piece_id=FPTLF2X6JMP7AKQ3IRR2ZGFGBR2N6CV4HG4CQKAQJESORRTY43PQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:01.237479027 (59h17m32.427777598s) [upload=56864]: Next returned nil segment Jul 26 22:53:01.237509085 (59h17m32.42780767s) [upload=56864]: Waiting for error group managing segments... Jul 26 22:53:01.388060980 (59h17m32.578359565s) [noise=true , address=79.127.219.38:4015 , piece_id=CEN5SHDZPFIR7ET3IQMZJ7XXMYHK452WT4NRPT6EYV5AOQ3JQIBQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:01.477019570 (59h17m32.667318385s) [noise=true , address=77.90.31.11:4564 , piece_id=VPM3YDZOHEQ2K4VGDFTSP2X5VB7GMSUHYPIC6NGNNZYHNCPZDX4Q, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:01.570426403 (59h17m32.760724994s) [noise=true , address=79.127.220.97:4361 , piece_id=UAC4MKLGHCW2T4CAZ44XPU6HFNV46XZ2VEHDO7RCGGUPUH5PAMLQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:02.443045146 (59h17m33.633343742s) [noise=true , address=79.127.205.234:4620 , piece_id=BETZASUWANWUZJMWNWBH4Q6F2FEEF3TVYPEFI7LJSWIU4TANWUEA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:02.553994078 (59h17m33.744292657s) [noise=true , address=45.91.116.8:4065 , piece_id=JJXC676BBRD3A34NQVHHRLQYACEURAQBGQIH4STVE3KRJSAAR34Q, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:03.797194512 (59h17m34.987493083s) [noise=true , address=64.49.13.140:4515 , piece_id=LFNDH2JP7BHTNQB4RFAJIGVI76F5KKZEWSVSCSXMRQP7WPWD5POA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:04.762522644 (59h17m35.952821225s) [noise=true , address=64.49.13.138:4482 , piece_id=NDC2CRBXQMAWY3S3GMR6IE6YZIJ6E3S6J4NR4ZTZG3YUOCWF3FUQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:05.113474688 (59h17m36.303773258s) [noise=true , address=79.127.219.47:4048 , piece_id=7QN4AGPILTMSZN7G5CHXPIOVUYBNFGLECPPIJCIKRLD7DSYLKBOQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:05.221436882 (59h17m36.411735455s) [noise=true , address=156.146.43.225:4449 , piece_id=2BLCW4PNLHMCTBKO3DDY2UWWGQ57R4GQYR5CJOJ4F3TKJQRHOOHA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:06.110375991 (59h17m37.300674561s) [noise=true , address=45.91.116.9:4267 , piece_id=DBRX7SPXRIPZWF3R336D3C4V5Y3RXTDOK7D5TRGELNCIGN2GJOGA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:07.366873835 (59h17m38.557172407s) [noise=true , address=77.90.31.16:4767 , piece_id=24TFO6NBPXF2B2734DTA4V3SISXM6IBBUVGL7IXHPZ5PBQJZJQRQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:09.150232925 (59h17m40.340531505s) [noise=true , address=45.91.116.11:4124 , piece_id=34S6H73HSOROLG4M3ZV3RR2TODJVF336O33VJDBRGGXSXXAJ7BHQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: <nil> Jul 26 22:53:09.300314202 (59h17m40.490612781s) [noise=true , address=45.91.116.14:4222 , piece_id=RCIFS6AAO6F24MDP5L3R4GLLP2ZPXV7KDJ6FIVVFOMJQRXVJ3VJA, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:00.192255385 (59h18m31.38255414s) [noise=true , address=77.90.31.18:4824 , piece_id=B6D7F7CUEBVMEPZ5SANZMDVT4QVJGC4RJSE2PVTYT7QOOGZEPLQQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 139.59.144.22:46048->77.90.31.18:4824: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:54:00.192355963 (59h18m31.382654543s) [noise=true , address=77.90.31.10:4528 , piece_id=T6XAYVQOZR2CP7YSNREVYRYHFJXH5LPKC43RNVEA5P7HLBXOSSYQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: write tcp 139.59.144.22:42790->77.90.31.10:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:54:00.192668569 (59h18m31.382967161s) [noise=true , address=77.90.31.17:4791 , piece_id=JCQDLV3LFITU3R3FVKCFF2Y7JRDIAO674NUON45SWN2EH4BTXGMQ, seg_pos={0 0}, upload=56864]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: read tcp 139.59.144.22:50080->77.90.31.17:4791: use of closed network connection; context deadline exceeded Jul 26 22:54:00.192755106 (59h18m31.383053703s) [seg_pos={0 0}, upload=56864]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:54:00.300405600 (59h18m31.490704167s) [upload=56864]: Done uploading segments... Jul 26 22:54:00.300432148 (59h18m31.490730727s) [upload=56864]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.110656089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 843.394011ms FAIL (2025-07-26 22:53:58 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 843.425567ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............................X.........] duration: 3.680661972s FAIL (2025-07-26 22:53:53 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 3.680711185s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.36896162s FAIL (2025-07-26 22:53:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.368991796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 872.301003ms FAIL (2025-07-26 22:53:14 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 872.330314ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............................X........] duration: 2.254274414s FAIL (2025-07-26 22:53:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.254309387s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [......X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.X] duration: 1m0.115484084s FAIL (2025-07-26 22:53:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 56849]: Uploading segments... Jul 26 22:52:00.224434869 (59h16m31.414733439s) [upload=56849]: Got next segment. Inline: false Jul 26 22:52:00.224499435 (59h16m31.414798005s) [upload=56849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:52:00.224505245 (59h16m31.414803815s) [upload=56849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:52:00.429440348 (59h16m31.619738935s) [seg_pos={0 0}, upload=56849]: Begin upload segment... Jul 26 22:52:00.429522494 (59h16m31.619821085s) [seg_pos={0 0}, upload=56849]: Done begin upload segment. Jul 26 22:52:00.429584588 (59h16m31.619883179s) [noise=true , address=45.91.116.14:4206 , piece_id=TUGQFJJVCXJ2L76W2KFW64DYSYGM4JBN4HKWPL4EYZC7FUMACMFQ, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.430069657 (59h16m31.620368247s) [noise=true , address=77.90.31.18:4839 , piece_id=HLKNX7SP4CVV53QERGWYPYC74H6HDS7VKFAYBCK3R5GJGB474QEA, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.430223577 (59h16m31.62052217s) [noise=true , address=77.90.31.10:4544 , piece_id=UTRRJ2NED46SLIACU6HJIZK65VBVG3LLVSTNCC7LA2JBHITZME5A, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.430547031 (59h16m31.62084562s) [noise=true , address=79.127.205.246:4279 , piece_id=2FSB2VYUCSQHIIT6IQQYSWE4EOWIMTJQN7A7I4HN6JY2W4ER2T6A, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.430768925 (59h16m31.621067515s) [noise=true , address=64.49.13.155:4918 , piece_id=II3HLPD4EW7YAQLJF5AN5Y6XBQDQQD2IIHQN3ZPFRVQTLJE4TZTA, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.430961159 (59h16m31.62125975s) [noise=true , address=79.127.205.233:4650 , piece_id=PELU6YCF6LJ6FZ5THUJ5BLFCHCCCVD7UPVCXIGGOZ6SAFW2QODGQ, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.431313314 (59h16m31.621611904s) [noise=true , address=79.127.205.245:4246 , piece_id=XCW2B377QVBLZRIOKXSVB6AZID37NUDUR6JGCC5D4J75GDNHPAAQ, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.431332546 (59h16m31.621631136s) [noise=true , address=79.127.220.97:4361 , piece_id=QDW5G4SG7ZP4MYDANQYIRFJQBGBX73UDOVTE4QF6QSQUW3E5AA3Q, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.431730958 (59h16m31.622029547s) [noise=true , address=64.49.13.170:4721 , piece_id=6AU5N6FKSPIWR7PCQSE36V3K7RPNXPA63SAMJT2UD356SKZN2EXQ, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.431807543 (59h16m31.622106133s) [noise=true , address=79.127.219.47:4061 , piece_id=XDAWII2OEIP7LWG3QLJD5OEFM6IMK4SIZFZNQEKM3GG5F4462FYA, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.432141338 (59h16m31.622439935s) [noise=true , address=156.146.43.225:4434 , piece_id=FJKALOX3L3BHSTPAF3VEDD35ZD42ZICEU3MAGLQEXYHQ635Q3TAQ, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.432336276 (59h16m31.622634859s) [noise=true , address=79.127.219.38:4009 , piece_id=PYHNENZZJBD3UOHESO6TFMPOI5WX2T7DRNMIK3OMFITRO6QQ45TA, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.432514226 (59h16m31.622812817s) [noise=true , address=79.127.219.48:4090 , piece_id=EBAGSPLBG3XBDWSUJVNT3QUGEEMQZJ6NDS2K7HTOBRQYQEVCOCSQ, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.433057278 (59h16m31.623355872s) [noise=true , address=79.127.219.51:4210 , piece_id=4R3HRYGXJWHWSD6R7IBNPQ6F343WLEK2BLVASGWIAJNNWWSS623A, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.433066915 (59h16m31.623365504s) [noise=true , address=45.91.116.9:4267 , piece_id=A4HNUW3EPSYDK5SS5CK4VTKAHAASVJXF44UMZ2CZ2XVQNPQDBMCA, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.433434352 (59h16m31.623732943s) [noise=true , address=79.127.205.234:4611 , piece_id=2CAYCEXIFGTSN5IOWPSSDM7VKO6BG5GLYP7IA52JKL2G5KRSMDBA, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.433515779 (59h16m31.623814369s) [noise=true , address=156.146.43.225:4449 , piece_id=5OWZY63UEI776F4ATW5VDZDFVT244HKXNPFWAZBM764SP4NNVG2A, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.433945684 (59h16m31.624244273s) [noise=true , address=77.90.31.16:4746 , piece_id=RDFORSGIXJZ3UYPK7ZFVMR3UO342FZEGL3MGCTGOEPXZNGAJR6OA, seg_pos={0 0}, upload=56849]: Uploading piece... Jul 26 22:52:00.821852656 (59h16m32.012151236s) [noise=true , address=77.90.31.16:4746 , piece_id=RDFORSGIXJZ3UYPK7ZFVMR3UO342FZEGL3MGCTGOEPXZNGAJR6OA, seg_pos={0 0}, upload=56849]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jul 26 22:52:00.934071287 (59h16m32.124369869s) [noise=true , address=156.146.43.225:4449 , piece_id=5OWZY63UEI776F4ATW5VDZDFVT244HKXNPFWAZBM764SP4NNVG2A, seg_pos={0 0}, upload=56849]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:52:00.953422459 (59h16m32.143721024s) [noise=true , address=79.127.219.48:4090 , piece_id=EBAGSPLBG3XBDWSUJVNT3QUGEEMQZJ6NDS2K7HTOBRQYQEVCOCSQ, seg_pos={0 0}, upload=56849]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:52:01.037732428 (59h16m32.228031012s) [upload=56849]: Next returned nil segment Jul 26 22:52:01.037755931 (59h16m32.228054522s) [upload=56849]: Waiting for error group managing segments... Jul 26 22:52:01.162024934 (59h16m32.352323524s) [noise=true , address=79.127.219.51:4210 , piece_id=4R3HRYGXJWHWSD6R7IBNPQ6F343WLEK2BLVASGWIAJNNWWSS623A, seg_pos={0 0}, upload=56849]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:52:01.199749889 (59h16m32.390048482s) [noise=true , address=79.127.219.38:4009 , piece_id=PYHNENZZJBD3UOHESO6TFMPOI5WX2T7DRNMIK3OMFITRO6QQ45TA, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:01.698209663 (59h16m32.888508231s) [noise=true , address=79.127.205.234:4611 , piece_id=2CAYCEXIFGTSN5IOWPSSDM7VKO6BG5GLYP7IA52JKL2G5KRSMDBA, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:01.727855246 (59h16m32.918153818s) [noise=true , address=79.127.205.246:4279 , piece_id=2FSB2VYUCSQHIIT6IQQYSWE4EOWIMTJQN7A7I4HN6JY2W4ER2T6A, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:01.757203692 (59h16m32.947502277s) [noise=true , address=79.127.205.245:4246 , piece_id=XCW2B377QVBLZRIOKXSVB6AZID37NUDUR6JGCC5D4J75GDNHPAAQ, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:01.847146726 (59h16m33.037445303s) [noise=true , address=64.49.13.155:4918 , piece_id=II3HLPD4EW7YAQLJF5AN5Y6XBQDQQD2IIHQN3ZPFRVQTLJE4TZTA, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:01.873874243 (59h16m33.064172824s) [noise=true , address=64.49.13.170:4721 , piece_id=6AU5N6FKSPIWR7PCQSE36V3K7RPNXPA63SAMJT2UD356SKZN2EXQ, seg_pos={0 0}, upload=56849]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:52:02.156869825 (59h16m33.347168395s) [noise=true , address=79.127.205.233:4650 , piece_id=PELU6YCF6LJ6FZ5THUJ5BLFCHCCCVD7UPVCXIGGOZ6SAFW2QODGQ, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:02.579390472 (59h16m33.769689058s) [noise=true , address=156.146.43.225:4434 , piece_id=FJKALOX3L3BHSTPAF3VEDD35ZD42ZICEU3MAGLQEXYHQ635Q3TAQ, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:03.933472535 (59h16m35.123771114s) [noise=true , address=79.127.219.47:4061 , piece_id=XDAWII2OEIP7LWG3QLJD5OEFM6IMK4SIZFZNQEKM3GG5F4462FYA, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:04.124115701 (59h16m35.31441427s) [noise=true , address=45.91.116.14:4206 , piece_id=TUGQFJJVCXJ2L76W2KFW64DYSYGM4JBN4HKWPL4EYZC7FUMACMFQ, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:04.359919016 (59h16m35.550217601s) [noise=true , address=79.127.220.97:4361 , piece_id=QDW5G4SG7ZP4MYDANQYIRFJQBGBX73UDOVTE4QF6QSQUW3E5AA3Q, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:05.246445741 (59h16m36.436744319s) [noise=true , address=45.91.116.9:4267 , piece_id=A4HNUW3EPSYDK5SS5CK4VTKAHAASVJXF44UMZ2CZ2XVQNPQDBMCA, seg_pos={0 0}, upload=56849]: Done uploading piece. err: <nil> Jul 26 22:52:59.939110131 (59h17m31.12940871s) [noise=true , address=77.90.31.10:4544 , piece_id=UTRRJ2NED46SLIACU6HJIZK65VBVG3LLVSTNCC7LA2JBHITZME5A, seg_pos={0 0}, upload=56849]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:57556->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:52:59.939965966 (59h17m31.130264546s) [noise=true , address=77.90.31.18:4839 , piece_id=HLKNX7SP4CVV53QERGWYPYC74H6HDS7VKFAYBCK3R5GJGB474QEA, seg_pos={0 0}, upload=56849]: Done uploading piece. err: ecclient: upload failed (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx, address:77.90.31.18:4839): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:52:59.940000253 (59h17m31.130298836s) [seg_pos={0 0}, upload=56849]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:53:00.052844811 (59h17m31.243143391s) [upload=56849]: Done uploading segments... Jul 26 22:53:00.052894842 (59h17m31.243193422s) [upload=56849]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.115511855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.261287224s FAIL (2025-07-26 22:52:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.261326042s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...............X.......X..X...X.XX...X..X.XX......X.X.......] duration: 9.805720964s FAIL (2025-07-26 22:52:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.80574328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............................X.......] duration: 2.303096236s FAIL (2025-07-26 22:52:32 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.303126397s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [................X.......X..X...X.XX...X..X.XX......X.X......] duration: 21.198306407s FAIL (2025-07-26 22:52:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.198332046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 807.396012ms FAIL (2025-07-26 22:52:13 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 807.421902ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.............................X......] duration: 2.317245264s FAIL (2025-07-26 22:52:06 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.317297311s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.361948976s FAIL (2025-07-26 22:51:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.361978864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration: 2.288413246s FAIL (2025-07-26 22:51:51 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.288452482s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [.......X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.] duration: 44.540499942s FAIL (2025-07-26 22:51:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ... Jul 26 22:49:44.561785392 (59h14m15.752083964s) [upload=56806]: Upload finished. err: <nil> Jul 26 22:49:55.451664472 (59h14m26.641963059s) [upload=56815]: Starting upload Jul 26 22:49:55.451748295 (59h14m26.642046864s) [upload=56815]: Done starting upload Jul 26 22:49:55.451961857 (59h14m26.642260425s) [upload=56815]: Uploading segments... Jul 26 22:49:55.452057184 (59h14m26.642355778s) [upload=56815]: Got next segment. Inline: false Jul 26 22:49:55.452115308 (59h14m26.642413889s) [upload=56815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:49:55.452123325 (59h14m26.642421905s) [upload=56815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:49:55.648335797 (59h14m26.838634387s) [seg_pos={0 0}, upload=56815]: Begin upload segment... Jul 26 22:49:55.648421592 (59h14m26.838720175s) [seg_pos={0 0}, upload=56815]: Done begin upload segment. Jul 26 22:49:55.648462417 (59h14m26.838761008s) [noise=true , address=77.90.31.11:4577 , piece_id=NNW47B4ZBOA2N2IDTE26X2MYBANSC354345BJI3SRSF2CCC3DKJQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.648487859 (59h14m26.838786445s) [noise=true , address=64.49.13.138:4483 , piece_id=W4TMRVPZ3CZM5HQQ7I64FP4XEG442XB6C24URPPC2DNX3BG3E4GQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649065696 (59h14m26.839364285s) [noise=true , address=79.127.219.38:4008 , piece_id=6Q7OHHRAEKATLMFJSJAHDASWYOMNAMRCS42C4LEUPSUPZI674FLA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649169223 (59h14m26.839467809s) [noise=true , address=77.90.31.18:4814 , piece_id=GAMQCPEBR6CRAD45LF44AGL7WVRHVGNP2AYJG23OUFOVA7O5WJFA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649374581 (59h14m26.839673162s) [noise=true , address=156.146.43.225:4449 , piece_id=PM55KQNCRWZ2L2C2PZ7HKNZSFJKRWAJC3MKDHIRWS4BUKHSGJTSA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649674715 (59h14m26.839973293s) [noise=true , address=79.127.205.245:4218 , piece_id=S3XANNFAJ4PNMUW75ZPBFVL3LIWR5LGBMJDCRBEMB7PWUMJZHFUA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649751859 (59h14m26.840050565s) [noise=true , address=45.91.116.11:4151 , piece_id=HTRQIQFMKHFACJ52P3JDPQX7TS6XCIUWA42FFOKCPG3ZT4SYB6RA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.650164126 (59h14m26.840462713s) [noise=true , address=45.91.116.14:4222 , piece_id=2VMRWAZOISMHSTUDFFSDYFQCJQZNFP7Z6DUXY4HEMAWCXX2VEVTA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.650588993 (59h14m26.84088758s) [noise=true , address=77.90.31.13:4644 , piece_id=4C2MYLSALKVDNCM5ML34R3VX7UGFN5LFDT7RAAKOHOXCQEN5JHSA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.650968738 (59h14m26.841267324s) [noise=true , address=79.127.205.233:4653 , piece_id=TPMO4LLQBZY2PVSMA7CG2BWJ3YB4N2NYTDVNFE23OFHSLHKGWTYA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.651359826 (59h14m26.841658415s) [noise=true , address=79.127.205.235:4687 , piece_id=JVFTQKGP673TKWLW7N5DOHEKOX6VSH7O43SDDVUJWTHHBQXAXCGQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.651794795 (59h14m26.842093385s) [noise=true , address=156.146.43.225:4434 , piece_id=SHLFQFRVOLKC42XWTA5ZNDHVJ2JHIYWJCZSBO3WDH72S3LSSQ6FQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.652180011 (59h14m26.8424786s) [noise=true , address=45.91.116.13:4175 , piece_id=3Z7SRMLRP4AXTZJ4AWM5EMPPZZ3GEDJ4JGLIWWMHKN72G5K7ALQA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.652454572 (59h14m26.842753162s) [noise=true , address=79.127.205.245:4247 , piece_id=X7HG2LYVD7ZTQTTNJL2VPZJO6BPJXGJLO6LMBZKR6MYTZSCILLMQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.652899392 (59h14m26.843197974s) [noise=true , address=79.127.219.47:4048 , piece_id=TTAY7H4JQVOVZF76VFROJJQFLFN2RLMIPXQYFMZYQSJH7VTEXXAA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.653763372 (59h14m26.844061961s) [noise=true , address=79.127.205.246:4272 , piece_id=GP6FJVIC2DWEHXD72Z4Y2E4LJLRLMOD666Y354BCDJCEKZITAITQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.654430161 (59h14m26.84472875s) [noise=true , address=45.91.116.14:4206 , piece_id=2QMMVEN7ZBSIEE4ZU2HI56OD2CSKDAXK54XHPGRSKZXLMSD3LIDQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.657755776 (59h14m26.848054368s) [noise=true , address=77.90.31.17:4800 , piece_id=YK2AYQSSODZPQGPUJE2UIGJLPZM3Z7ZLAN66K3EFBAVWH6AYQSNA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:56.067136003 (59h14m27.257434572s) [noise=true , address=79.127.219.47:4048 , piece_id=TTAY7H4JQVOVZF76VFROJJQFLFN2RLMIPXQYFMZYQSJH7VTEXXAA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:56.171161523 (59h14m27.361460102s) [noise=true , address=156.146.43.225:4449 , piece_id=PM55KQNCRWZ2L2C2PZ7HKNZSFJKRWAJC3MKDHIRWS4BUKHSGJTSA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:56.223829968 (59h14m27.414128555s) [upload=56815]: Next returned nil segment Jul 26 22:49:56.223851539 (59h14m27.41415011s) [upload=56815]: Waiting for error group managing segments... Jul 26 22:49:56.382129796 (59h14m27.572428384s) [noise=true , address=156.146.43.225:4434 , piece_id=SHLFQFRVOLKC42XWTA5ZNDHVJ2JHIYWJCZSBO3WDH72S3LSSQ6FQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:56.417381120 (59h14m27.607679685s) [noise=true , address=77.90.31.11:4577 , piece_id=NNW47B4ZBOA2N2IDTE26X2MYBANSC354345BJI3SRSF2CCC3DKJQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:56.903108986 (59h14m28.093407556s) [noise=true , address=79.127.205.246:4272 , piece_id=GP6FJVIC2DWEHXD72Z4Y2E4LJLRLMOD666Y354BCDJCEKZITAITQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:56.937528750 (59h14m28.127827315s) [noise=true , address=79.127.205.245:4218 , piece_id=S3XANNFAJ4PNMUW75ZPBFVL3LIWR5LGBMJDCRBEMB7PWUMJZHFUA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.031875274 (59h14m28.22217384s) [noise=true , address=79.127.219.38:4008 , piece_id=6Q7OHHRAEKATLMFJSJAHDASWYOMNAMRCS42C4LEUPSUPZI674FLA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.034171963 (59h14m28.224470548s) [noise=true , address=79.127.205.245:4247 , piece_id=X7HG2LYVD7ZTQTTNJL2VPZJO6BPJXGJLO6LMBZKR6MYTZSCILLMQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.174602813 (59h14m28.364901381s) [noise=true , address=79.127.205.233:4653 , piece_id=TPMO4LLQBZY2PVSMA7CG2BWJ3YB4N2NYTDVNFE23OFHSLHKGWTYA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.322137560 (59h14m28.512436156s) [noise=true , address=79.127.205.235:4687 , piece_id=JVFTQKGP673TKWLW7N5DOHEKOX6VSH7O43SDDVUJWTHHBQXAXCGQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:58.832473097 (59h14m30.022771684s) [noise=true , address=45.91.116.14:4206 , piece_id=2QMMVEN7ZBSIEE4ZU2HI56OD2CSKDAXK54XHPGRSKZXLMSD3LIDQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:00.289082658 (59h14m31.479381237s) [noise=true , address=45.91.116.13:4175 , piece_id=3Z7SRMLRP4AXTZJ4AWM5EMPPZZ3GEDJ4JGLIWWMHKN72G5K7ALQA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:01.797840951 (59h14m32.988139522s) [noise=true , address=64.49.13.138:4483 , piece_id=W4TMRVPZ3CZM5HQQ7I64FP4XEG442XB6C24URPPC2DNX3BG3E4GQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:10.058779830 (59h14m41.249078411s) [noise=true , address=45.91.116.11:4151 , piece_id=HTRQIQFMKHFACJ52P3JDPQX7TS6XCIUWA42FFOKCPG3ZT4SYB6RA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:10.147750460 (59h14m41.338049029s) [noise=true , address=45.91.116.14:4222 , piece_id=2VMRWAZOISMHSTUDFFSDYFQCJQZNFP7Z6DUXY4HEMAWCXX2VEVTA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:11.826704763 (59h14m43.017003336s) [noise=true , address=77.90.31.13:4644 , piece_id=4C2MYLSALKVDNCM5ML34R3VX7UGFN5LFDT7RAAKOHOXCQEN5JHSA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:55.183055002 (59h15m26.373353586s) [noise=true , address=77.90.31.17:4800 , piece_id=YK2AYQSSODZPQGPUJE2UIGJLPZM3Z7ZLAN66K3EFBAVWH6AYQSNA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:55.183109172 (59h15m26.373407752s) [noise=true , address=77.90.31.18:4814 , piece_id=GAMQCPEBR6CRAD45LF44AGL7WVRHVGNP2AYJG23OUFOVA7O5WJFA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: read tcp 139.59.144.22:58086->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:50:55.183139344 (59h15m26.37343791s) [seg_pos={0 0}, upload=56815]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:50:55.292791322 (59h15m26.483089908s) [upload=56815]: Done uploading segments... Jul 26 22:50:55.292828343 (59h15m26.483126932s) [upload=56815]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 44.540529228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.5414373s FAIL (2025-07-26 22:51:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.541469597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.................X.......X..X...X.XX...X..X.XX......X.X.....] duration: 23.514053626s FAIL (2025-07-26 22:51:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.514080308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 888.530805ms FAIL (2025-07-26 22:51:15 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 888.563511ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 910.544901ms FAIL (2025-07-26 22:50:58 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 910.572971ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [........X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X] duration: 1m0.112051756s FAIL (2025-07-26 22:50:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ... Jul 26 22:49:44.561785392 (59h14m15.752083964s) [upload=56806]: Upload finished. err: <nil> Jul 26 22:49:55.451664472 (59h14m26.641963059s) [upload=56815]: Starting upload Jul 26 22:49:55.451748295 (59h14m26.642046864s) [upload=56815]: Done starting upload Jul 26 22:49:55.451961857 (59h14m26.642260425s) [upload=56815]: Uploading segments... Jul 26 22:49:55.452057184 (59h14m26.642355778s) [upload=56815]: Got next segment. Inline: false Jul 26 22:49:55.452115308 (59h14m26.642413889s) [upload=56815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:49:55.452123325 (59h14m26.642421905s) [upload=56815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:49:55.648335797 (59h14m26.838634387s) [seg_pos={0 0}, upload=56815]: Begin upload segment... Jul 26 22:49:55.648421592 (59h14m26.838720175s) [seg_pos={0 0}, upload=56815]: Done begin upload segment. Jul 26 22:49:55.648462417 (59h14m26.838761008s) [noise=true , address=77.90.31.11:4577 , piece_id=NNW47B4ZBOA2N2IDTE26X2MYBANSC354345BJI3SRSF2CCC3DKJQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.648487859 (59h14m26.838786445s) [noise=true , address=64.49.13.138:4483 , piece_id=W4TMRVPZ3CZM5HQQ7I64FP4XEG442XB6C24URPPC2DNX3BG3E4GQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649065696 (59h14m26.839364285s) [noise=true , address=79.127.219.38:4008 , piece_id=6Q7OHHRAEKATLMFJSJAHDASWYOMNAMRCS42C4LEUPSUPZI674FLA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649169223 (59h14m26.839467809s) [noise=true , address=77.90.31.18:4814 , piece_id=GAMQCPEBR6CRAD45LF44AGL7WVRHVGNP2AYJG23OUFOVA7O5WJFA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649374581 (59h14m26.839673162s) [noise=true , address=156.146.43.225:4449 , piece_id=PM55KQNCRWZ2L2C2PZ7HKNZSFJKRWAJC3MKDHIRWS4BUKHSGJTSA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649674715 (59h14m26.839973293s) [noise=true , address=79.127.205.245:4218 , piece_id=S3XANNFAJ4PNMUW75ZPBFVL3LIWR5LGBMJDCRBEMB7PWUMJZHFUA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.649751859 (59h14m26.840050565s) [noise=true , address=45.91.116.11:4151 , piece_id=HTRQIQFMKHFACJ52P3JDPQX7TS6XCIUWA42FFOKCPG3ZT4SYB6RA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.650164126 (59h14m26.840462713s) [noise=true , address=45.91.116.14:4222 , piece_id=2VMRWAZOISMHSTUDFFSDYFQCJQZNFP7Z6DUXY4HEMAWCXX2VEVTA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.650588993 (59h14m26.84088758s) [noise=true , address=77.90.31.13:4644 , piece_id=4C2MYLSALKVDNCM5ML34R3VX7UGFN5LFDT7RAAKOHOXCQEN5JHSA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.650968738 (59h14m26.841267324s) [noise=true , address=79.127.205.233:4653 , piece_id=TPMO4LLQBZY2PVSMA7CG2BWJ3YB4N2NYTDVNFE23OFHSLHKGWTYA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.651359826 (59h14m26.841658415s) [noise=true , address=79.127.205.235:4687 , piece_id=JVFTQKGP673TKWLW7N5DOHEKOX6VSH7O43SDDVUJWTHHBQXAXCGQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.651794795 (59h14m26.842093385s) [noise=true , address=156.146.43.225:4434 , piece_id=SHLFQFRVOLKC42XWTA5ZNDHVJ2JHIYWJCZSBO3WDH72S3LSSQ6FQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.652180011 (59h14m26.8424786s) [noise=true , address=45.91.116.13:4175 , piece_id=3Z7SRMLRP4AXTZJ4AWM5EMPPZZ3GEDJ4JGLIWWMHKN72G5K7ALQA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.652454572 (59h14m26.842753162s) [noise=true , address=79.127.205.245:4247 , piece_id=X7HG2LYVD7ZTQTTNJL2VPZJO6BPJXGJLO6LMBZKR6MYTZSCILLMQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.652899392 (59h14m26.843197974s) [noise=true , address=79.127.219.47:4048 , piece_id=TTAY7H4JQVOVZF76VFROJJQFLFN2RLMIPXQYFMZYQSJH7VTEXXAA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.653763372 (59h14m26.844061961s) [noise=true , address=79.127.205.246:4272 , piece_id=GP6FJVIC2DWEHXD72Z4Y2E4LJLRLMOD666Y354BCDJCEKZITAITQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.654430161 (59h14m26.84472875s) [noise=true , address=45.91.116.14:4206 , piece_id=2QMMVEN7ZBSIEE4ZU2HI56OD2CSKDAXK54XHPGRSKZXLMSD3LIDQ, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:55.657755776 (59h14m26.848054368s) [noise=true , address=77.90.31.17:4800 , piece_id=YK2AYQSSODZPQGPUJE2UIGJLPZM3Z7ZLAN66K3EFBAVWH6AYQSNA, seg_pos={0 0}, upload=56815]: Uploading piece... Jul 26 22:49:56.067136003 (59h14m27.257434572s) [noise=true , address=79.127.219.47:4048 , piece_id=TTAY7H4JQVOVZF76VFROJJQFLFN2RLMIPXQYFMZYQSJH7VTEXXAA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:56.171161523 (59h14m27.361460102s) [noise=true , address=156.146.43.225:4449 , piece_id=PM55KQNCRWZ2L2C2PZ7HKNZSFJKRWAJC3MKDHIRWS4BUKHSGJTSA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:56.223829968 (59h14m27.414128555s) [upload=56815]: Next returned nil segment Jul 26 22:49:56.223851539 (59h14m27.41415011s) [upload=56815]: Waiting for error group managing segments... Jul 26 22:49:56.382129796 (59h14m27.572428384s) [noise=true , address=156.146.43.225:4434 , piece_id=SHLFQFRVOLKC42XWTA5ZNDHVJ2JHIYWJCZSBO3WDH72S3LSSQ6FQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:56.417381120 (59h14m27.607679685s) [noise=true , address=77.90.31.11:4577 , piece_id=NNW47B4ZBOA2N2IDTE26X2MYBANSC354345BJI3SRSF2CCC3DKJQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:56.903108986 (59h14m28.093407556s) [noise=true , address=79.127.205.246:4272 , piece_id=GP6FJVIC2DWEHXD72Z4Y2E4LJLRLMOD666Y354BCDJCEKZITAITQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:56.937528750 (59h14m28.127827315s) [noise=true , address=79.127.205.245:4218 , piece_id=S3XANNFAJ4PNMUW75ZPBFVL3LIWR5LGBMJDCRBEMB7PWUMJZHFUA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.031875274 (59h14m28.22217384s) [noise=true , address=79.127.219.38:4008 , piece_id=6Q7OHHRAEKATLMFJSJAHDASWYOMNAMRCS42C4LEUPSUPZI674FLA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.034171963 (59h14m28.224470548s) [noise=true , address=79.127.205.245:4247 , piece_id=X7HG2LYVD7ZTQTTNJL2VPZJO6BPJXGJLO6LMBZKR6MYTZSCILLMQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.174602813 (59h14m28.364901381s) [noise=true , address=79.127.205.233:4653 , piece_id=TPMO4LLQBZY2PVSMA7CG2BWJ3YB4N2NYTDVNFE23OFHSLHKGWTYA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:57.322137560 (59h14m28.512436156s) [noise=true , address=79.127.205.235:4687 , piece_id=JVFTQKGP673TKWLW7N5DOHEKOX6VSH7O43SDDVUJWTHHBQXAXCGQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:49:58.832473097 (59h14m30.022771684s) [noise=true , address=45.91.116.14:4206 , piece_id=2QMMVEN7ZBSIEE4ZU2HI56OD2CSKDAXK54XHPGRSKZXLMSD3LIDQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:00.289082658 (59h14m31.479381237s) [noise=true , address=45.91.116.13:4175 , piece_id=3Z7SRMLRP4AXTZJ4AWM5EMPPZZ3GEDJ4JGLIWWMHKN72G5K7ALQA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:01.797840951 (59h14m32.988139522s) [noise=true , address=64.49.13.138:4483 , piece_id=W4TMRVPZ3CZM5HQQ7I64FP4XEG442XB6C24URPPC2DNX3BG3E4GQ, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:10.058779830 (59h14m41.249078411s) [noise=true , address=45.91.116.11:4151 , piece_id=HTRQIQFMKHFACJ52P3JDPQX7TS6XCIUWA42FFOKCPG3ZT4SYB6RA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:10.147750460 (59h14m41.338049029s) [noise=true , address=45.91.116.14:4222 , piece_id=2VMRWAZOISMHSTUDFFSDYFQCJQZNFP7Z6DUXY4HEMAWCXX2VEVTA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:11.826704763 (59h14m43.017003336s) [noise=true , address=77.90.31.13:4644 , piece_id=4C2MYLSALKVDNCM5ML34R3VX7UGFN5LFDT7RAAKOHOXCQEN5JHSA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: <nil> Jul 26 22:50:55.183055002 (59h15m26.373353586s) [noise=true , address=77.90.31.17:4800 , piece_id=YK2AYQSSODZPQGPUJE2UIGJLPZM3Z7ZLAN66K3EFBAVWH6AYQSNA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:55.183109172 (59h15m26.373407752s) [noise=true , address=77.90.31.18:4814 , piece_id=GAMQCPEBR6CRAD45LF44AGL7WVRHVGNP2AYJG23OUFOVA7O5WJFA, seg_pos={0 0}, upload=56815]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: read tcp 139.59.144.22:58086->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:50:55.183139344 (59h15m26.37343791s) [seg_pos={0 0}, upload=56815]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:50:55.292791322 (59h15m26.483089908s) [upload=56815]: Done uploading segments... Jul 26 22:50:55.292828343 (59h15m26.483126932s) [upload=56815]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.112081865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...............X...........................................] duration: 2.283827733s FAIL (2025-07-26 22:50:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.283939662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............................X....] duration: 2.604228641s FAIL (2025-07-26 22:50:50 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.604267874s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 839.134354ms FAIL (2025-07-26 22:50:37 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 839.166346ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...............X..........................................] duration: 2.302715669s FAIL (2025-07-26 22:50:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.302743956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 858.128175ms FAIL (2025-07-26 22:50:21 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 858.160997ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..................X.......X..X...X.XX...X..X.XX......X.X....] duration: 11.515561441s FAIL (2025-07-26 22:50:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.515581341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...................X.......X..X...X.XX...X..X.XX......X.X...] duration: 17.658092782s FAIL (2025-07-26 22:50:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.658122848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 918.886199ms FAIL (2025-07-26 22:50:02 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 918.935704ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X...............X.........................................] duration: 2.243740875s FAIL (2025-07-26 22:49:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.243787054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [.........X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.] duration: 16.161392734s FAIL (2025-07-26 22:49:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 05640 (59h13m4.552704232s) [upload=56790]: Got next segment. Inline: false Jul 26 22:48:33.362487843 (59h13m4.552786431s) [upload=56790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:48:33.362495290 (59h13m4.552793873s) [upload=56790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:48:33.557341009 (59h13m4.747639589s) [seg_pos={0 0}, upload=56790]: Begin upload segment... Jul 26 22:48:33.557402742 (59h13m4.747701312s) [seg_pos={0 0}, upload=56790]: Done begin upload segment. Jul 26 22:48:33.557445597 (59h13m4.747744178s) [noise=true , address=156.146.43.225:4434 , piece_id=YBNIPSXYV6CHYGTWPWEWUQHGZUWAWB2CTH2C737SX4B4UKYR7IBA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.557465476 (59h13m4.747764053s) [noise=true , address=79.127.219.51:4201 , piece_id=BNEOJI6DCTCR4G4DZXNYZKLX77A7G4KZIBNDHP444SPFIVX4ITOA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.557828679 (59h13m4.748127261s) [noise=true , address=79.127.220.97:4361 , piece_id=M5FOCLSBXDXGLLIE5SHQQL67P64N2IYIXEZI7WSIYKDTPWBOTMGQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.558169705 (59h13m4.74846829s) [noise=true , address=77.90.31.13:4644 , piece_id=4B6WCS63CY35PDFFO5DOSNBLZA6F5SOPI4TJIJK5PO3A5NJF6N7A, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.559947497 (59h13m4.750246077s) [noise=true , address=77.90.31.10:4538 , piece_id=MDORMW4GCOBZWWRX327J5FCZE5INPXUXNFX3PB3IMFBAKGTHTQVA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.560928295 (59h13m4.751226874s) [noise=true , address=79.127.219.47:4061 , piece_id=OAUPJFOS4KEC32WBUX746DQW7YUYHF3W7NH4TZDCV2ATAJ6BJVRQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.561346626 (59h13m4.751645208s) [noise=true , address=45.91.116.14:4206 , piece_id=K3O25WK4ZI4JZCBPHY54X5UL7MRM4R5O5U4ITKU4BBAFQANWF64A, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.561699846 (59h13m4.751998432s) [noise=true , address=79.127.219.51:4212 , piece_id=LMSGZS4CWN7UJUK52DYVJZSRN7C3356BER4P45EUVZH7EQ2HJNEQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.562119943 (59h13m4.752418521s) [noise=true , address=77.90.31.21:4907 , piece_id=XH4KAZF33JRXJYKVVEAWIFM6LDGFTPKYFUN6OOCGQH2ZQZSA3GNA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.562428956 (59h13m4.752727538s) [noise=true , address=79.127.205.245:4228 , piece_id=IGZ73ODKYRCXRGHCH6LX5BX2DSN3OBVWE73QGRSHAV7TD2NO75CA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.562681993 (59h13m4.752980578s) [noise=true , address=64.49.13.140:4514 , piece_id=5YNWXTXSZ24M2XKWNVOMRWOFJHUEWKOG3RIIWBYPJL33E53BJDBQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.563111216 (59h13m4.753409804s) [noise=true , address=45.91.116.8:4065 , piece_id=L7E5U4CPYJLKZG63K7B2VRAKCZKJSRH5IU7VWJDILGS552JPD3LQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.563469802 (59h13m4.75376839s) [noise=true , address=79.127.205.246:4272 , piece_id=V4FPK52E35MT6C5WLNZBIIEZVB64VDFGOUQ4VQIDOYMWKI23PHZA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.563791947 (59h13m4.754090527s) [noise=true , address=156.146.43.225:4449 , piece_id=A4PWGYNLITVO2ZJUT42VZFVOLI36BXXBALLYCQJ4WLD7DGYQNYAA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.564216888 (59h13m4.754515473s) [noise=true , address=45.91.116.11:4121 , piece_id=G2JDJYMQ2F2GC5K5IVQMGF5T2IIDGQBXP73ZLGI6L5Z3GJLIFTCQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.564730785 (59h13m4.755029373s) [noise=true , address=64.49.13.170:4721 , piece_id=7KNG6RJR4PKEEL57F3YWAHBMFHXCECPY6IEC7SHDL3W6QYNOTCKA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.567672226 (59h13m4.757970806s) [noise=true , address=79.127.220.101:4048 , piece_id=3MBMDPZVLYER3EVJTOPA7V43TSLNRSDKFDZOARSXR4AHOTMXAMBA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.569041011 (59h13m4.7593396s) [noise=true , address=77.90.31.17:4785 , piece_id=O2XK7KRG6I4BZ5SSJMEQWNBHUI7DAQTOC6I6LTQGKMBZPEDJLMJQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:34.110698022 (59h13m5.3009966s) [noise=true , address=156.146.43.225:4434 , piece_id=YBNIPSXYV6CHYGTWPWEWUQHGZUWAWB2CTH2C737SX4B4UKYR7IBA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:34.117793363 (59h13m5.308091933s) [noise=true , address=156.146.43.225:4449 , piece_id=A4PWGYNLITVO2ZJUT42VZFVOLI36BXXBALLYCQJ4WLD7DGYQNYAA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:34.146893736 (59h13m5.337192315s) [upload=56790]: Next returned nil segment Jul 26 22:48:34.146938538 (59h13m5.337237119s) [upload=56790]: Waiting for error group managing segments... Jul 26 22:48:34.321825587 (59h13m5.512124159s) [noise=true , address=77.90.31.21:4907 , piece_id=XH4KAZF33JRXJYKVVEAWIFM6LDGFTPKYFUN6OOCGQH2ZQZSA3GNA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.370789378 (59h13m5.561087944s) [noise=true , address=45.91.116.11:4121 , piece_id=G2JDJYMQ2F2GC5K5IVQMGF5T2IIDGQBXP73ZLGI6L5Z3GJLIFTCQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.804472280 (59h13m5.994770863s) [noise=true , address=79.127.219.51:4212 , piece_id=LMSGZS4CWN7UJUK52DYVJZSRN7C3356BER4P45EUVZH7EQ2HJNEQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.836613673 (59h13m6.026912245s) [noise=true , address=79.127.205.246:4272 , piece_id=V4FPK52E35MT6C5WLNZBIIEZVB64VDFGOUQ4VQIDOYMWKI23PHZA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.937173241 (59h13m6.127471815s) [noise=true , address=79.127.205.245:4228 , piece_id=IGZ73ODKYRCXRGHCH6LX5BX2DSN3OBVWE73QGRSHAV7TD2NO75CA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:35.392734487 (59h13m6.583033056s) [noise=true , address=64.49.13.140:4514 , piece_id=5YNWXTXSZ24M2XKWNVOMRWOFJHUEWKOG3RIIWBYPJL33E53BJDBQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:35.492493771 (59h13m6.682792358s) [noise=true , address=79.127.219.51:4201 , piece_id=BNEOJI6DCTCR4G4DZXNYZKLX77A7G4KZIBNDHP444SPFIVX4ITOA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:35.841471603 (59h13m7.03177017s) [noise=true , address=64.49.13.170:4721 , piece_id=7KNG6RJR4PKEEL57F3YWAHBMFHXCECPY6IEC7SHDL3W6QYNOTCKA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:36.739261465 (59h13m7.929560045s) [noise=true , address=45.91.116.14:4206 , piece_id=K3O25WK4ZI4JZCBPHY54X5UL7MRM4R5O5U4ITKU4BBAFQANWF64A, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:37.151614377 (59h13m8.341912942s) [noise=true , address=79.127.219.47:4061 , piece_id=OAUPJFOS4KEC32WBUX746DQW7YUYHF3W7NH4TZDCV2ATAJ6BJVRQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:37.495865094 (59h13m8.686163927s) [noise=true , address=45.91.116.8:4065 , piece_id=L7E5U4CPYJLKZG63K7B2VRAKCZKJSRH5IU7VWJDILGS552JPD3LQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:38.366975273 (59h13m9.557273855s) [noise=true , address=79.127.220.97:4361 , piece_id=M5FOCLSBXDXGLLIE5SHQQL67P64N2IYIXEZI7WSIYKDTPWBOTMGQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:39.949248108 (59h13m11.139546681s) [noise=true , address=79.127.220.101:4048 , piece_id=3MBMDPZVLYER3EVJTOPA7V43TSLNRSDKFDZOARSXR4AHOTMXAMBA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:41.330807829 (59h13m12.521106409s) [noise=true , address=77.90.31.13:4644 , piece_id=4B6WCS63CY35PDFFO5DOSNBLZA6F5SOPI4TJIJK5PO3A5NJF6N7A, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:33.094089111 (59h14m4.284387691s) [noise=true , address=77.90.31.10:4538 , piece_id=MDORMW4GCOBZWWRX327J5FCZE5INPXUXNFX3PB3IMFBAKGTHTQVA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: write tcp 139.59.144.22:53014->77.90.31.10:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:49:33.094128204 (59h14m4.284426775s) [noise=true , address=77.90.31.17:4785 , piece_id=O2XK7KRG6I4BZ5SSJMEQWNBHUI7DAQTOC6I6LTQGKMBZPEDJLMJQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN, address:77.90.31.17:4785): protocol: write tcp 139.59.144.22:49514->77.90.31.17:4785: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:49:33.094191633 (59h14m4.284490213s) [seg_pos={0 0}, upload=56790]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:49:33.199716716 (59h14m4.390015282s) [upload=56790]: Done uploading segments... Jul 26 22:49:33.199743991 (59h14m4.390042556s) [upload=56790]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 16.161415966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.X...............X........................................] duration: 2.267821417s FAIL (2025-07-26 22:49:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.267918675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.........................X.............................X...] duration: 2.383524355s FAIL (2025-07-26 22:49:39 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.383557566s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [..........X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X] duration: 1m0.10822295s FAIL (2025-07-26 22:49:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 05640 (59h13m4.552704232s) [upload=56790]: Got next segment. Inline: false Jul 26 22:48:33.362487843 (59h13m4.552786431s) [upload=56790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:48:33.362495290 (59h13m4.552793873s) [upload=56790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:48:33.557341009 (59h13m4.747639589s) [seg_pos={0 0}, upload=56790]: Begin upload segment... Jul 26 22:48:33.557402742 (59h13m4.747701312s) [seg_pos={0 0}, upload=56790]: Done begin upload segment. Jul 26 22:48:33.557445597 (59h13m4.747744178s) [noise=true , address=156.146.43.225:4434 , piece_id=YBNIPSXYV6CHYGTWPWEWUQHGZUWAWB2CTH2C737SX4B4UKYR7IBA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.557465476 (59h13m4.747764053s) [noise=true , address=79.127.219.51:4201 , piece_id=BNEOJI6DCTCR4G4DZXNYZKLX77A7G4KZIBNDHP444SPFIVX4ITOA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.557828679 (59h13m4.748127261s) [noise=true , address=79.127.220.97:4361 , piece_id=M5FOCLSBXDXGLLIE5SHQQL67P64N2IYIXEZI7WSIYKDTPWBOTMGQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.558169705 (59h13m4.74846829s) [noise=true , address=77.90.31.13:4644 , piece_id=4B6WCS63CY35PDFFO5DOSNBLZA6F5SOPI4TJIJK5PO3A5NJF6N7A, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.559947497 (59h13m4.750246077s) [noise=true , address=77.90.31.10:4538 , piece_id=MDORMW4GCOBZWWRX327J5FCZE5INPXUXNFX3PB3IMFBAKGTHTQVA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.560928295 (59h13m4.751226874s) [noise=true , address=79.127.219.47:4061 , piece_id=OAUPJFOS4KEC32WBUX746DQW7YUYHF3W7NH4TZDCV2ATAJ6BJVRQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.561346626 (59h13m4.751645208s) [noise=true , address=45.91.116.14:4206 , piece_id=K3O25WK4ZI4JZCBPHY54X5UL7MRM4R5O5U4ITKU4BBAFQANWF64A, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.561699846 (59h13m4.751998432s) [noise=true , address=79.127.219.51:4212 , piece_id=LMSGZS4CWN7UJUK52DYVJZSRN7C3356BER4P45EUVZH7EQ2HJNEQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.562119943 (59h13m4.752418521s) [noise=true , address=77.90.31.21:4907 , piece_id=XH4KAZF33JRXJYKVVEAWIFM6LDGFTPKYFUN6OOCGQH2ZQZSA3GNA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.562428956 (59h13m4.752727538s) [noise=true , address=79.127.205.245:4228 , piece_id=IGZ73ODKYRCXRGHCH6LX5BX2DSN3OBVWE73QGRSHAV7TD2NO75CA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.562681993 (59h13m4.752980578s) [noise=true , address=64.49.13.140:4514 , piece_id=5YNWXTXSZ24M2XKWNVOMRWOFJHUEWKOG3RIIWBYPJL33E53BJDBQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.563111216 (59h13m4.753409804s) [noise=true , address=45.91.116.8:4065 , piece_id=L7E5U4CPYJLKZG63K7B2VRAKCZKJSRH5IU7VWJDILGS552JPD3LQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.563469802 (59h13m4.75376839s) [noise=true , address=79.127.205.246:4272 , piece_id=V4FPK52E35MT6C5WLNZBIIEZVB64VDFGOUQ4VQIDOYMWKI23PHZA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.563791947 (59h13m4.754090527s) [noise=true , address=156.146.43.225:4449 , piece_id=A4PWGYNLITVO2ZJUT42VZFVOLI36BXXBALLYCQJ4WLD7DGYQNYAA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.564216888 (59h13m4.754515473s) [noise=true , address=45.91.116.11:4121 , piece_id=G2JDJYMQ2F2GC5K5IVQMGF5T2IIDGQBXP73ZLGI6L5Z3GJLIFTCQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.564730785 (59h13m4.755029373s) [noise=true , address=64.49.13.170:4721 , piece_id=7KNG6RJR4PKEEL57F3YWAHBMFHXCECPY6IEC7SHDL3W6QYNOTCKA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.567672226 (59h13m4.757970806s) [noise=true , address=79.127.220.101:4048 , piece_id=3MBMDPZVLYER3EVJTOPA7V43TSLNRSDKFDZOARSXR4AHOTMXAMBA, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:33.569041011 (59h13m4.7593396s) [noise=true , address=77.90.31.17:4785 , piece_id=O2XK7KRG6I4BZ5SSJMEQWNBHUI7DAQTOC6I6LTQGKMBZPEDJLMJQ, seg_pos={0 0}, upload=56790]: Uploading piece... Jul 26 22:48:34.110698022 (59h13m5.3009966s) [noise=true , address=156.146.43.225:4434 , piece_id=YBNIPSXYV6CHYGTWPWEWUQHGZUWAWB2CTH2C737SX4B4UKYR7IBA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:34.117793363 (59h13m5.308091933s) [noise=true , address=156.146.43.225:4449 , piece_id=A4PWGYNLITVO2ZJUT42VZFVOLI36BXXBALLYCQJ4WLD7DGYQNYAA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:34.146893736 (59h13m5.337192315s) [upload=56790]: Next returned nil segment Jul 26 22:48:34.146938538 (59h13m5.337237119s) [upload=56790]: Waiting for error group managing segments... Jul 26 22:48:34.321825587 (59h13m5.512124159s) [noise=true , address=77.90.31.21:4907 , piece_id=XH4KAZF33JRXJYKVVEAWIFM6LDGFTPKYFUN6OOCGQH2ZQZSA3GNA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.370789378 (59h13m5.561087944s) [noise=true , address=45.91.116.11:4121 , piece_id=G2JDJYMQ2F2GC5K5IVQMGF5T2IIDGQBXP73ZLGI6L5Z3GJLIFTCQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.804472280 (59h13m5.994770863s) [noise=true , address=79.127.219.51:4212 , piece_id=LMSGZS4CWN7UJUK52DYVJZSRN7C3356BER4P45EUVZH7EQ2HJNEQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.836613673 (59h13m6.026912245s) [noise=true , address=79.127.205.246:4272 , piece_id=V4FPK52E35MT6C5WLNZBIIEZVB64VDFGOUQ4VQIDOYMWKI23PHZA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:34.937173241 (59h13m6.127471815s) [noise=true , address=79.127.205.245:4228 , piece_id=IGZ73ODKYRCXRGHCH6LX5BX2DSN3OBVWE73QGRSHAV7TD2NO75CA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:35.392734487 (59h13m6.583033056s) [noise=true , address=64.49.13.140:4514 , piece_id=5YNWXTXSZ24M2XKWNVOMRWOFJHUEWKOG3RIIWBYPJL33E53BJDBQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:35.492493771 (59h13m6.682792358s) [noise=true , address=79.127.219.51:4201 , piece_id=BNEOJI6DCTCR4G4DZXNYZKLX77A7G4KZIBNDHP444SPFIVX4ITOA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:35.841471603 (59h13m7.03177017s) [noise=true , address=64.49.13.170:4721 , piece_id=7KNG6RJR4PKEEL57F3YWAHBMFHXCECPY6IEC7SHDL3W6QYNOTCKA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:36.739261465 (59h13m7.929560045s) [noise=true , address=45.91.116.14:4206 , piece_id=K3O25WK4ZI4JZCBPHY54X5UL7MRM4R5O5U4ITKU4BBAFQANWF64A, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:37.151614377 (59h13m8.341912942s) [noise=true , address=79.127.219.47:4061 , piece_id=OAUPJFOS4KEC32WBUX746DQW7YUYHF3W7NH4TZDCV2ATAJ6BJVRQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:37.495865094 (59h13m8.686163927s) [noise=true , address=45.91.116.8:4065 , piece_id=L7E5U4CPYJLKZG63K7B2VRAKCZKJSRH5IU7VWJDILGS552JPD3LQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:38.366975273 (59h13m9.557273855s) [noise=true , address=79.127.220.97:4361 , piece_id=M5FOCLSBXDXGLLIE5SHQQL67P64N2IYIXEZI7WSIYKDTPWBOTMGQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:39.949248108 (59h13m11.139546681s) [noise=true , address=79.127.220.101:4048 , piece_id=3MBMDPZVLYER3EVJTOPA7V43TSLNRSDKFDZOARSXR4AHOTMXAMBA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: <nil> Jul 26 22:48:41.330807829 (59h13m12.521106409s) [noise=true , address=77.90.31.13:4644 , piece_id=4B6WCS63CY35PDFFO5DOSNBLZA6F5SOPI4TJIJK5PO3A5NJF6N7A, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:33.094089111 (59h14m4.284387691s) [noise=true , address=77.90.31.10:4538 , piece_id=MDORMW4GCOBZWWRX327J5FCZE5INPXUXNFX3PB3IMFBAKGTHTQVA, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: write tcp 139.59.144.22:53014->77.90.31.10:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:49:33.094128204 (59h14m4.284426775s) [noise=true , address=77.90.31.17:4785 , piece_id=O2XK7KRG6I4BZ5SSJMEQWNBHUI7DAQTOC6I6LTQGKMBZPEDJLMJQ, seg_pos={0 0}, upload=56790]: Done uploading piece. err: ecclient: upload failed (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN, address:77.90.31.17:4785): protocol: write tcp 139.59.144.22:49514->77.90.31.17:4785: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:49:33.094191633 (59h14m4.284490213s) [seg_pos={0 0}, upload=56790]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:49:33.199716716 (59h14m4.390015282s) [upload=56790]: Done uploading segments... Jul 26 22:49:33.199743991 (59h14m4.390042556s) [upload=56790]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.108319685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 881.416953ms FAIL (2025-07-26 22:49:16 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 881.453762ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)