Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.086337868s FAIL (2025-07-26 23:05:04 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 1.08644163s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 824.951063ms FAIL (2025-07-26 23:04:28 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 825.100532ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.........XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXXX] duration: 1m0.114044073s FAIL (2025-07-26 23:04:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== ent. Inline: false Jul 26 23:03:18.188681710 (59h27m49.378980289s) [upload=57026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:03:18.188709977 (59h27m49.379008556s) [upload=57026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:03:18.382795352 (59h27m49.573093931s) [seg_pos={0 0}, upload=57026]: Begin upload segment... Jul 26 23:03:18.382850870 (59h27m49.573149448s) [seg_pos={0 0}, upload=57026]: Done begin upload segment. Jul 26 23:03:18.382879073 (59h27m49.573177655s) [noise=true , address=45.91.116.4:4943 , piece_id=HHK5S7GUMLQZLONMSLBBWVQTQQYTNTYRSDXIHIUU6SRPDOTTMPRQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.382925521 (59h27m49.573224104s) [noise=true , address=79.127.205.233:4647 , piece_id=NYGVQ2O37H2VDBXJ6SH57FPHGKHGHQ7QARLI6L33APJX2OZ7CDPA, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.383236210 (59h27m49.573534789s) [noise=true , address=77.90.31.12:4617 , piece_id=UDRCJEG3EO47WO4JI2GYTWENKVLMAK2PVLIBFYJCRSKBMWK4F3ZA, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.383496984 (59h27m49.573795563s) [noise=true , address=79.127.220.97:4361 , piece_id=DQSU3OJCUAOEBAY3EQJKVIS4ZLWTKYKHIA3S34R45X5GSKY6VYIQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.383549400 (59h27m49.573847986s) [noise=true , address=79.127.219.51:4210 , piece_id=5FJQMVNEMNHY725PQA6VHKONXXDEDZAYE2HNKODMRSZ5FE2D2TZQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.383734226 (59h27m49.574032806s) [noise=true , address=79.127.205.246:4280 , piece_id=Z2Z4STNDNVDGXF65WKEKYW6O4Y3I52YLTS6B4Z3LACRT35VNKKOQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.383970567 (59h27m49.574269146s) [noise=true , address=79.127.205.245:4249 , piece_id=4A344MVOT7TAQZRTUHRL4RT3TGJESI3UPZFX4E5IWJ2BPWN6UKIQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.384286441 (59h27m49.574585037s) [noise=true , address=64.49.13.140:4514 , piece_id=QPVQQ3534MKXPGRSNAZY3HNPKDATK2M3MP2VJ2BQZKTWSJ4YIPSA, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.384676411 (59h27m49.574974991s) [noise=true , address=77.90.31.15:4712 , piece_id=7BVTEA5OLKUOECXMT5YWD2EJOBO4DJH4FIOTTKUKOI5EXXETESVQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.384979003 (59h27m49.575277592s) [noise=true , address=79.127.201.215:4351 , piece_id=AJ5G4DPJUICZSR7SHSED5V7ZETJSVKOJQ2JOOC7DP4TMC22GK4ZQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.385208564 (59h27m49.57550715s) [noise=true , address=77.90.31.11:4587 , piece_id=ZTSF6JJXZWK2WAAIMVWLAHHBLO7JKA6X6A2TYZ7L7MINVBB7NDIA, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.385404064 (59h27m49.575702646s) [noise=true , address=79.127.201.214:4321 , piece_id=3FGDRALY2RFGGPNZ43KZNKV5ZYXQ5KFOECXK3H6BPVKXK2LT4X6A, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.385719628 (59h27m49.576018215s) [noise=true , address=77.90.31.21:4908 , piece_id=I6VOFRV3G32UKMUQPXLYXBYM65S3773SIGYO5NIVAGOFDRWSX4AQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.386041921 (59h27m49.576340503s) [noise=true , address=79.127.201.215:4358 , piece_id=MZ6EEOODT6MA5K2F6OYJXPSOLYLLFB6CQMO4ZLWB6FZSICDG7SZA, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.386298687 (59h27m49.576597269s) [noise=true , address=79.127.219.51:4212 , piece_id=A2Z3OJVAUYSVGKE2MDW6TE2LFTCERPHW4NEYNKWM7HF4VBLPW6TQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.386535630 (59h27m49.576834211s) [noise=true , address=79.127.205.235:4669 , piece_id=EW3QOAQAXELP2FEJXQDPOAU3AMNY6YHYXE45I2QGOBXMVUOCCTOA, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.386700221 (59h27m49.576998802s) [noise=true , address=45.91.116.14:4206 , piece_id=FEEDOFQFH4C2WTBKOLAPXVXUOTUSNT4OXS3VQ2UFLQU24N6JHYSQ, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.387004203 (59h27m49.577302773s) [noise=true , address=79.127.220.101:4048 , piece_id=LOZPMU23KTU7F23UN7U66TCCLH2DVYKJF2QTGSK4XLYRPQXI4XIA, seg_pos={0 0}, upload=57026]: Uploading piece... Jul 26 23:03:18.807572738 (59h27m49.997871307s) [noise=true , address=79.127.219.51:4212 , piece_id=A2Z3OJVAUYSVGKE2MDW6TE2LFTCERPHW4NEYNKWM7HF4VBLPW6TQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:18.888621986 (59h27m50.078920567s) [noise=true , address=79.127.219.51:4210 , piece_id=5FJQMVNEMNHY725PQA6VHKONXXDEDZAYE2HNKODMRSZ5FE2D2TZQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:18.922091684 (59h27m50.112390268s) [noise=true , address=79.127.220.101:4048 , piece_id=LOZPMU23KTU7F23UN7U66TCCLH2DVYKJF2QTGSK4XLYRPQXI4XIA, seg_pos={0 0}, upload=57026]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:18.954862973 (59h27m50.145161542s) [upload=57026]: Next returned nil segment Jul 26 23:03:18.954882419 (59h27m50.145180989s) [upload=57026]: Waiting for error group managing segments... Jul 26 23:03:19.130565986 (59h27m50.320864551s) [noise=true , address=77.90.31.21:4908 , piece_id=I6VOFRV3G32UKMUQPXLYXBYM65S3773SIGYO5NIVAGOFDRWSX4AQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:19.361640596 (59h27m50.551939186s) [noise=true , address=64.49.13.140:4514 , piece_id=QPVQQ3534MKXPGRSNAZY3HNPKDATK2M3MP2VJ2BQZKTWSJ4YIPSA, seg_pos={0 0}, upload=57026]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 23:03:19.616135033 (59h27m50.806433606s) [noise=true , address=79.127.205.246:4280 , piece_id=Z2Z4STNDNVDGXF65WKEKYW6O4Y3I52YLTS6B4Z3LACRT35VNKKOQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:19.674250458 (59h27m50.864549023s) [noise=true , address=79.127.205.233:4647 , piece_id=NYGVQ2O37H2VDBXJ6SH57FPHGKHGHQ7QARLI6L33APJX2OZ7CDPA, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:19.722757498 (59h27m50.913056089s) [noise=true , address=79.127.205.245:4249 , piece_id=4A344MVOT7TAQZRTUHRL4RT3TGJESI3UPZFX4E5IWJ2BPWN6UKIQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:19.827412765 (59h27m51.017711353s) [noise=true , address=79.127.205.235:4669 , piece_id=EW3QOAQAXELP2FEJXQDPOAU3AMNY6YHYXE45I2QGOBXMVUOCCTOA, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:21.654658829 (59h27m52.84495741s) [noise=true , address=79.127.220.97:4361 , piece_id=DQSU3OJCUAOEBAY3EQJKVIS4ZLWTKYKHIA3S34R45X5GSKY6VYIQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:22.310422741 (59h27m53.500721313s) [noise=true , address=79.127.201.214:4321 , piece_id=3FGDRALY2RFGGPNZ43KZNKV5ZYXQ5KFOECXK3H6BPVKXK2LT4X6A, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:25.797735013 (59h27m56.988033583s) [noise=true , address=45.91.116.14:4206 , piece_id=FEEDOFQFH4C2WTBKOLAPXVXUOTUSNT4OXS3VQ2UFLQU24N6JHYSQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:03:27.318135977 (59h27m58.508434548s) [noise=true , address=45.91.116.4:4943 , piece_id=HHK5S7GUMLQZLONMSLBBWVQTQQYTNTYRSDXIHIUU6SRPDOTTMPRQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: ecclient: upload failed (node:128QuaUBgwagbp1fCbMDLMncgCth8bd4kaMTnd4ebP1pvQMB1jv, address:45.91.116.4:4943): protocol: not enough available disk space, have: 0, need: 6725632 Jul 26 23:03:57.321358084 (59h28m28.511656649s) [noise=true , address=77.90.31.12:4617 , piece_id=UDRCJEG3EO47WO4JI2GYTWENKVLMAK2PVLIBFYJCRSKBMWK4F3ZA, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:04:03.741588281 (59h28m34.931886858s) [noise=true , address=79.127.201.215:4358 , piece_id=MZ6EEOODT6MA5K2F6OYJXPSOLYLLFB6CQMO4ZLWB6FZSICDG7SZA, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:04:06.776827196 (59h28m37.967125765s) [noise=true , address=79.127.201.215:4351 , piece_id=AJ5G4DPJUICZSR7SHSED5V7ZETJSVKOJQ2JOOC7DP4TMC22GK4ZQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: <nil> Jul 26 23:04:17.914841282 (59h28m49.105139852s) [noise=true , address=77.90.31.11:4587 , piece_id=ZTSF6JJXZWK2WAAIMVWLAHHBLO7JKA6X6A2TYZ7L7MINVBB7NDIA, seg_pos={0 0}, upload=57026]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: write tcp 139.59.144.22:58304->77.90.31.11:4587: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:04:17.914936695 (59h28m49.105235299s) [noise=true , address=77.90.31.15:4712 , piece_id=7BVTEA5OLKUOECXMT5YWD2EJOBO4DJH4FIOTTKUKOI5EXXETESVQ, seg_pos={0 0}, upload=57026]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: write tcp 139.59.144.22:55616->77.90.31.15:4712: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:04:17.915011597 (59h28m49.105310187s) [seg_pos={0 0}, upload=57026]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:04:18.027103453 (59h28m49.217402026s) [upload=57026]: Done uploading segments... Jul 26 23:04:18.027130964 (59h28m49.217429543s) [upload=57026]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.114064819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.............................X..................X...] duration: 2.25957557s FAIL (2025-07-26 23:04:17 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.259611073s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.296960151s FAIL (2025-07-26 23:04:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 280->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:44.058547940 (59h24m15.24884652s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:59:44.058641678 (59h24m15.248940257s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:59:44.058646789 (59h24m15.248945359s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:59:44.224851647 (59h24m15.415150235s) [upload=56969]: Done uploading segments... Jul 26 22:59:44.224876852 (59h24m15.415175436s) [upload=56969]: Upload finished. err: <nil> Jul 26 23:00:00.341276866 (59h24m31.531575453s) [upload=56972]: Starting upload Jul 26 23:00:00.341377270 (59h24m31.531675865s) [upload=56972]: Done starting upload Jul 26 23:00:00.341505637 (59h24m31.531804221s) [upload=56972]: Uploading segments... Jul 26 23:00:00.341940351 (59h24m31.532238944s) [upload=56972]: Got next segment. Inline: false Jul 26 23:00:00.342010661 (59h24m31.532309252s) [upload=56972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:00.342017759 (59h24m31.532316348s) [upload=56972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:00.455411486 (59h24m31.64571009s) [seg_pos={0 0}, upload=56972]: Begin upload segment... Jul 26 23:00:00.455493514 (59h24m31.645792104s) [seg_pos={0 0}, upload=56972]: Done begin upload segment. Jul 26 23:00:00.455540024 (59h24m31.645838613s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.455561207 (59h24m31.645859795s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456074349 (59h24m31.646372943s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456243971 (59h24m31.64654256s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456605120 (59h24m31.646903743s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456973436 (59h24m31.647272026s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457050035 (59h24m31.647348625s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457398570 (59h24m31.64769716s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457455587 (59h24m31.647754178s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457779166 (59h24m31.648077758s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458115180 (59h24m31.648413769s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458194315 (59h24m31.648492905s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458592315 (59h24m31.648890905s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458651549 (59h24m31.648950139s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.459145371 (59h24m31.649443962s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.462726202 (59h24m31.653024793s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:01.079541782 (59h24m32.269840354s) [upload=56972]: Next returned nil segment Jul 26 23:00:01.079566166 (59h24m32.269864784s) [upload=56972]: Waiting for error group managing segments... Jul 26 23:00:01.210918643 (59h24m32.401217229s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.228724770 (59h24m32.419023355s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232110801 (59h24m32.422409369s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232305605 (59h24m32.422604172s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234390772 (59h24m32.424689363s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234614195 (59h24m32.424912769s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246294708 (59h24m32.43659329s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246456357 (59h24m32.43675493s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.247086738 (59h24m32.437385324s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.248804736 (59h24m32.43910332s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.302224542 (59h24m32.492523129s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:30.042884997 (59h25m1.233183578s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:48592->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043116623 (59h25m1.233415194s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043228739 (59h25m1.233527306s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043315635 (59h25m1.233614206s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043325410 (59h25m1.233623995s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:50064->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043373737 (59h25m1.23367233s) [seg_pos={0 0}, upload=56972]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:00:30.159169528 (59h25m1.349468101s) [upload=56972]: Done uploading segments... Jul 26 23:00:30.159218327 (59h25m1.34951691s) [upload=56972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.29704767s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..X.......X..X...X.XX...X..X.XX......X.X.......X.......X....] duration: 13.235220031s FAIL (2025-07-26 23:03:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.235256134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...X.......X..X...X.XX...X..X.XX......X.X.......X.......X...] duration: 48.384581135s FAIL (2025-07-26 23:03:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.384672189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.389981965s FAIL (2025-07-26 23:03:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 280->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:44.058547940 (59h24m15.24884652s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:59:44.058641678 (59h24m15.248940257s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:59:44.058646789 (59h24m15.248945359s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:59:44.224851647 (59h24m15.415150235s) [upload=56969]: Done uploading segments... Jul 26 22:59:44.224876852 (59h24m15.415175436s) [upload=56969]: Upload finished. err: <nil> Jul 26 23:00:00.341276866 (59h24m31.531575453s) [upload=56972]: Starting upload Jul 26 23:00:00.341377270 (59h24m31.531675865s) [upload=56972]: Done starting upload Jul 26 23:00:00.341505637 (59h24m31.531804221s) [upload=56972]: Uploading segments... Jul 26 23:00:00.341940351 (59h24m31.532238944s) [upload=56972]: Got next segment. Inline: false Jul 26 23:00:00.342010661 (59h24m31.532309252s) [upload=56972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:00.342017759 (59h24m31.532316348s) [upload=56972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:00.455411486 (59h24m31.64571009s) [seg_pos={0 0}, upload=56972]: Begin upload segment... Jul 26 23:00:00.455493514 (59h24m31.645792104s) [seg_pos={0 0}, upload=56972]: Done begin upload segment. Jul 26 23:00:00.455540024 (59h24m31.645838613s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.455561207 (59h24m31.645859795s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456074349 (59h24m31.646372943s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456243971 (59h24m31.64654256s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456605120 (59h24m31.646903743s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456973436 (59h24m31.647272026s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457050035 (59h24m31.647348625s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457398570 (59h24m31.64769716s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457455587 (59h24m31.647754178s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457779166 (59h24m31.648077758s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458115180 (59h24m31.648413769s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458194315 (59h24m31.648492905s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458592315 (59h24m31.648890905s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458651549 (59h24m31.648950139s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.459145371 (59h24m31.649443962s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.462726202 (59h24m31.653024793s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:01.079541782 (59h24m32.269840354s) [upload=56972]: Next returned nil segment Jul 26 23:00:01.079566166 (59h24m32.269864784s) [upload=56972]: Waiting for error group managing segments... Jul 26 23:00:01.210918643 (59h24m32.401217229s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.228724770 (59h24m32.419023355s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232110801 (59h24m32.422409369s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232305605 (59h24m32.422604172s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234390772 (59h24m32.424689363s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234614195 (59h24m32.424912769s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246294708 (59h24m32.43659329s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246456357 (59h24m32.43675493s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.247086738 (59h24m32.437385324s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.248804736 (59h24m32.43910332s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.302224542 (59h24m32.492523129s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:30.042884997 (59h25m1.233183578s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:48592->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043116623 (59h25m1.233415194s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043228739 (59h25m1.233527306s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043315635 (59h25m1.233614206s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043325410 (59h25m1.233623995s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:50064->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043373737 (59h25m1.23367233s) [seg_pos={0 0}, upload=56972]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:00:30.159169528 (59h25m1.349468101s) [upload=56972]: Done uploading segments... Jul 26 23:00:30.159218327 (59h25m1.34951691s) [upload=56972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.390013994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 829.96131ms FAIL (2025-07-26 23:03:19 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 829.994705ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [..........XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXXX] duration: 1m0.116698583s FAIL (2025-07-26 23:03:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d enough pieces (needed at least 14 but got 13) Jul 26 23:02:17.931463341 (59h26m49.12176255s) [upload=57010]: Starting upload Jul 26 23:02:17.931562954 (59h26m49.121861543s) [upload=57010]: Done starting upload Jul 26 23:02:17.931788361 (59h26m49.122086937s) [upload=57010]: Uploading segments... Jul 26 23:02:17.932117927 (59h26m49.122416508s) [upload=57010]: Got next segment. Inline: false Jul 26 23:02:17.932175980 (59h26m49.122474561s) [upload=57010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:02:17.932199223 (59h26m49.122497812s) [upload=57010]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:02:18.128936547 (59h26m49.319235142s) [seg_pos={0 0}, upload=57010]: Begin upload segment... Jul 26 23:02:18.129017853 (59h26m49.319316423s) [seg_pos={0 0}, upload=57010]: Done begin upload segment. Jul 26 23:02:18.129051353 (59h26m49.319349941s) [noise=true , address=77.90.31.11:4576 , piece_id=NQX56FZMVJMMOIUN4KOEVPXJODGUKRVA7UB3NSCBAWSI2FR2DALA, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.129649627 (59h26m49.319948213s) [noise=true , address=79.127.205.235:4683 , piece_id=7ODHMBFM6AOEN5DWIKWCISGTBGVM5U2F6KCQ3DBQDPXAIKMBICIQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.130073210 (59h26m49.320371793s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IZELGCLE2MP2B6VRXARYEECFMM6GVGLSQTXUKL2FMN7QM7RNLQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.130099186 (59h26m49.320397772s) [noise=true , address=64.49.13.155:4923 , piece_id=VOHCUJIPARZPQD47CU6GDGRNJROZZYTPCZUOWW7X6CQZ4XHAY2NQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.130517479 (59h26m49.320816071s) [noise=true , address=64.49.13.170:4718 , piece_id=E2MQAQ4P6KGR7XALORRURW62OEMNZ7TYJSYIJQARWXQ25NQ6Y6RA, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.130552259 (59h26m49.320850842s) [noise=true , address=79.127.219.47:4048 , piece_id=GC3FQXIRODJISKUVJ4ZT4VC7N4LP5N6SATBGKLECD74XDO6LHASQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.130946684 (59h26m49.321245273s) [noise=true , address=79.127.205.246:4258 , piece_id=6FNHD22CVHGAQ637ZP6H7O4RXI2JX5OB4FGEHPS5TLDBAMEDBH2Q, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.131071987 (59h26m49.321370576s) [noise=true , address=77.90.31.15:4727 , piece_id=MF2DYZ2FQKWFQOGWR35QUK3A6IPYVGV3MN2GCH5BT6MLSNM4QDHA, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.131358796 (59h26m49.32165738s) [noise=true , address=79.127.201.214:4321 , piece_id=F2KRGFJ4VTZLY3PNCZKTDKUH57HNITH22QUFHNNMR6OPJOFRBAUQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.131547204 (59h26m49.321845792s) [noise=true , address=45.91.116.9:4273 , piece_id=T3J7D7U4DLVHOYDB7TF7QFCG3KISWIUWSGT4FGHCIIHJ3JOP57UQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.131749189 (59h26m49.322047779s) [noise=true , address=45.91.116.14:4194 , piece_id=KR6A5PXPGAG6ROK2N2HJ7L7EPGIN3EE5MIBKPQSDQBTLUPXQYSFA, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.131936197 (59h26m49.322234782s) [noise=true , address=79.127.205.245:4226 , piece_id=VXV3J6CLTJEBTYRABDR24QZEJPH22RHQL4BLAOBEVUUQ6IUSQZOQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.132112485 (59h26m49.32241107s) [noise=true , address=79.127.219.38:4003 , piece_id=73F3IBL7XM57QY46SE74EHIC4FN5MTRL4RHSQMZLXJPNRDP3P3RQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.132418791 (59h26m49.322717378s) [noise=true , address=156.146.43.225:4434 , piece_id=6VMK773A2EHZFO4RBFHU5KK57D6WAXRI2DJLLCRIDZVEC3EU5K3A, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.133730374 (59h26m49.324028954s) [noise=true , address=77.90.31.12:4611 , piece_id=PUQEB67CXCPRFB7G6H346K7YGDFZBKMLJNPAXZZ5RQNP3RNB3B2A, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.134483615 (59h26m49.324782203s) [noise=true , address=79.127.205.246:4288 , piece_id=XHHYUBA3COMJHQ6NB3ATF6HDX6FQOBTSKAHGMNUX4WUBTJZJ2PJQ, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.135149441 (59h26m49.32544802s) [noise=true , address=64.49.13.140:4514 , piece_id=PJ5OBKZ6THRNS635T7ITE3QNUKRDG6RHTMCLRT4TH2R2Y423OK3Q, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.135915933 (59h26m49.32621452s) [noise=true , address=79.127.220.101:4048 , piece_id=TL735FGJXFEGDAYLSPMMWJPNN6LQ45FJ4M7OK2Y4WIJ6FVQHHB5Q, seg_pos={0 0}, upload=57010]: Uploading piece... Jul 26 23:02:18.525735187 (59h26m49.71603377s) [noise=true , address=79.127.219.47:4048 , piece_id=GC3FQXIRODJISKUVJ4ZT4VC7N4LP5N6SATBGKLECD74XDO6LHASQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:18.701788395 (59h26m49.892086968s) [upload=57010]: Next returned nil segment Jul 26 23:02:18.701808864 (59h26m49.892107443s) [upload=57010]: Waiting for error group managing segments... Jul 26 23:02:18.873892242 (59h26m50.064190812s) [noise=true , address=45.91.116.9:4273 , piece_id=T3J7D7U4DLVHOYDB7TF7QFCG3KISWIUWSGT4FGHCIIHJ3JOP57UQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:18.990050888 (59h26m50.180349456s) [noise=true , address=45.91.116.14:4194 , piece_id=KR6A5PXPGAG6ROK2N2HJ7L7EPGIN3EE5MIBKPQSDQBTLUPXQYSFA, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:19.038514353 (59h26m50.22881292s) [noise=true , address=79.127.219.38:4003 , piece_id=73F3IBL7XM57QY46SE74EHIC4FN5MTRL4RHSQMZLXJPNRDP3P3RQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:19.408607358 (59h26m50.598905927s) [noise=true , address=79.127.205.235:4683 , piece_id=7ODHMBFM6AOEN5DWIKWCISGTBGVM5U2F6KCQ3DBQDPXAIKMBICIQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:19.439968144 (59h26m50.630266724s) [noise=true , address=79.127.205.246:4258 , piece_id=6FNHD22CVHGAQ637ZP6H7O4RXI2JX5OB4FGEHPS5TLDBAMEDBH2Q, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:19.462690096 (59h26m50.652988669s) [noise=true , address=79.127.205.245:4226 , piece_id=VXV3J6CLTJEBTYRABDR24QZEJPH22RHQL4BLAOBEVUUQ6IUSQZOQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:19.519272567 (59h26m50.709571135s) [noise=true , address=77.90.31.12:4611 , piece_id=PUQEB67CXCPRFB7G6H346K7YGDFZBKMLJNPAXZZ5RQNP3RNB3B2A, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:19.625985743 (59h26m50.816284324s) [noise=true , address=79.127.205.246:4288 , piece_id=XHHYUBA3COMJHQ6NB3ATF6HDX6FQOBTSKAHGMNUX4WUBTJZJ2PJQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:20.222110539 (59h26m51.41240911s) [noise=true , address=64.49.13.140:4514 , piece_id=PJ5OBKZ6THRNS635T7ITE3QNUKRDG6RHTMCLRT4TH2R2Y423OK3Q, seg_pos={0 0}, upload=57010]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:20.232094355 (59h26m51.422392941s) [noise=true , address=79.127.220.101:4048 , piece_id=TL735FGJXFEGDAYLSPMMWJPNN6LQ45FJ4M7OK2Y4WIJ6FVQHHB5Q, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:21.232047697 (59h26m52.422346285s) [noise=true , address=79.127.201.214:4321 , piece_id=F2KRGFJ4VTZLY3PNCZKTDKUH57HNITH22QUFHNNMR6OPJOFRBAUQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:21.522778393 (59h26m52.713076972s) [noise=true , address=79.127.220.97:4361 , piece_id=S6IZELGCLE2MP2B6VRXARYEECFMM6GVGLSQTXUKL2FMN7QM7RNLQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:22.439963031 (59h26m53.630261613s) [noise=true , address=156.146.43.225:4434 , piece_id=6VMK773A2EHZFO4RBFHU5KK57D6WAXRI2DJLLCRIDZVEC3EU5K3A, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:02:24.238693098 (59h26m55.428991685s) [noise=true , address=64.49.13.155:4923 , piece_id=VOHCUJIPARZPQD47CU6GDGRNJROZZYTPCZUOWW7X6CQZ4XHAY2NQ, seg_pos={0 0}, upload=57010]: Done uploading piece. err: ecclient: upload failed (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z, address:64.49.13.155:4923): protocol: storage node overloaded, request limit: 2000 Jul 26 23:02:28.086631582 (59h26m59.276930163s) [noise=true , address=64.49.13.170:4718 , piece_id=E2MQAQ4P6KGR7XALORRURW62OEMNZ7TYJSYIJQARWXQ25NQ6Y6RA, seg_pos={0 0}, upload=57010]: Done uploading piece. err: <nil> Jul 26 23:03:17.661874407 (59h27m48.852172986s) [noise=true , address=77.90.31.15:4727 , piece_id=MF2DYZ2FQKWFQOGWR35QUK3A6IPYVGV3MN2GCH5BT6MLSNM4QDHA, seg_pos={0 0}, upload=57010]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:17.662030667 (59h27m48.85232925s) [noise=true , address=77.90.31.11:4576 , piece_id=NQX56FZMVJMMOIUN4KOEVPXJODGUKRVA7UB3NSCBAWSI2FR2DALA, seg_pos={0 0}, upload=57010]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: write tcp 139.59.144.22:43786->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:03:17.662055376 (59h27m48.85235396s) [seg_pos={0 0}, upload=57010]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:03:17.776739713 (59h27m48.967038292s) [upload=57010]: Done uploading segments... Jul 26 23:03:17.776769229 (59h27m48.967067819s) [upload=57010]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.116723591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.............................X..................X..] duration: 2.275507267s FAIL (2025-07-26 23:03:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.275540764s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.............................X..................X.] duration: 4.331643749s FAIL (2025-07-26 23:02:42 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 4.331672129s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 833.430058ms FAIL (2025-07-26 23:02:23 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 833.461827ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....X.......X..X...X.XX...X..X.XX......X.X.......X.......X..] duration: 10.85131297s FAIL (2025-07-26 23:02:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.851345845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................................X...] duration: 2.286793186s FAIL (2025-07-26 23:02:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 280->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:44.058547940 (59h24m15.24884652s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:59:44.058641678 (59h24m15.248940257s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:59:44.058646789 (59h24m15.248945359s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:59:44.224851647 (59h24m15.415150235s) [upload=56969]: Done uploading segments... Jul 26 22:59:44.224876852 (59h24m15.415175436s) [upload=56969]: Upload finished. err: <nil> Jul 26 23:00:00.341276866 (59h24m31.531575453s) [upload=56972]: Starting upload Jul 26 23:00:00.341377270 (59h24m31.531675865s) [upload=56972]: Done starting upload Jul 26 23:00:00.341505637 (59h24m31.531804221s) [upload=56972]: Uploading segments... Jul 26 23:00:00.341940351 (59h24m31.532238944s) [upload=56972]: Got next segment. Inline: false Jul 26 23:00:00.342010661 (59h24m31.532309252s) [upload=56972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:00.342017759 (59h24m31.532316348s) [upload=56972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:00.455411486 (59h24m31.64571009s) [seg_pos={0 0}, upload=56972]: Begin upload segment... Jul 26 23:00:00.455493514 (59h24m31.645792104s) [seg_pos={0 0}, upload=56972]: Done begin upload segment. Jul 26 23:00:00.455540024 (59h24m31.645838613s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.455561207 (59h24m31.645859795s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456074349 (59h24m31.646372943s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456243971 (59h24m31.64654256s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456605120 (59h24m31.646903743s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456973436 (59h24m31.647272026s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457050035 (59h24m31.647348625s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457398570 (59h24m31.64769716s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457455587 (59h24m31.647754178s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457779166 (59h24m31.648077758s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458115180 (59h24m31.648413769s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458194315 (59h24m31.648492905s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458592315 (59h24m31.648890905s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458651549 (59h24m31.648950139s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.459145371 (59h24m31.649443962s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.462726202 (59h24m31.653024793s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:01.079541782 (59h24m32.269840354s) [upload=56972]: Next returned nil segment Jul 26 23:00:01.079566166 (59h24m32.269864784s) [upload=56972]: Waiting for error group managing segments... Jul 26 23:00:01.210918643 (59h24m32.401217229s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.228724770 (59h24m32.419023355s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232110801 (59h24m32.422409369s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232305605 (59h24m32.422604172s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234390772 (59h24m32.424689363s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234614195 (59h24m32.424912769s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246294708 (59h24m32.43659329s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246456357 (59h24m32.43675493s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.247086738 (59h24m32.437385324s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.248804736 (59h24m32.43910332s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.302224542 (59h24m32.492523129s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:30.042884997 (59h25m1.233183578s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:48592->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043116623 (59h25m1.233415194s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043228739 (59h25m1.233527306s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043315635 (59h25m1.233614206s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043325410 (59h25m1.233623995s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:50064->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043373737 (59h25m1.23367233s) [seg_pos={0 0}, upload=56972]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:00:30.159169528 (59h25m1.349468101s) [upload=56972]: Done uploading segments... Jul 26 23:00:30.159218327 (59h25m1.34951691s) [upload=56972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.286828511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [...........XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXXX] duration: 1m0.115469633s FAIL (2025-07-26 23:02:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ) Jul 26 23:01:09.772384401 (59h25m40.962682983s) [upload=56993]: Starting upload Jul 26 23:01:09.772642444 (59h25m40.962941024s) [upload=56993]: Done starting upload Jul 26 23:01:09.772742628 (59h25m40.963041208s) [upload=56993]: Uploading segments... Jul 26 23:01:09.773077767 (59h25m40.963376347s) [upload=56993]: Got next segment. Inline: false Jul 26 23:01:09.773132531 (59h25m40.963431111s) [upload=56993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:01:09.773139109 (59h25m40.963437689s) [upload=56993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:01:09.991846182 (59h25m41.182144762s) [seg_pos={0 0}, upload=56993]: Begin upload segment... Jul 26 23:01:09.991897306 (59h25m41.182195879s) [seg_pos={0 0}, upload=56993]: Done begin upload segment. Jul 26 23:01:09.991923718 (59h25m41.1822223s) [noise=true , address=79.127.205.245:4252 , piece_id=4WRSOFZ2YH5ATAGFW4KSM3TGVPFFJUB3GCURDYLUSXCGJ2WHGEIA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.992340198 (59h25m41.182638783s) [noise=true , address=79.127.219.38:4007 , piece_id=MU2WCMJ6NWIS5Y3SIXUBH3CNO3L4OHDYQ7UGGF7IGZ6SJAMEAS3Q, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.992354496 (59h25m41.182653077s) [noise=true , address=77.90.31.12:4617 , piece_id=7LPFNSFFYRFWJSC5VK357I56RVWCI6QESSPDLNEAUQOSGJKPXGPQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.992818164 (59h25m41.183116749s) [noise=true , address=79.127.219.48:4090 , piece_id=AQTJGH6G2IYVSXJX3PGFWN3DPWKSWHKM5OFM7XD5SWRPUS2TD6VQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.992970896 (59h25m41.183269475s) [noise=true , address=45.91.116.14:4222 , piece_id=M62GI27NNODLKM5LVFG5D4CECC72IORZI23ODYJ4PU3GANMNI3JA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.993170919 (59h25m41.183469498s) [noise=true , address=79.127.219.51:4210 , piece_id=CKKBXLR4OTH2NCV355TZEMUMOT7FHELV4ZITVJV4M5UWK72JJWKA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.993372755 (59h25m41.183671341s) [noise=true , address=77.90.31.11:4585 , piece_id=Y7YAKWFFFHW7HARAQDGOMKPZDQLXC72SD6FNO73YYOJGW4CTNGJQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.993464793 (59h25m41.183763372s) [noise=true , address=79.127.219.38:4018 , piece_id=Q5UZCBJYPR6X67A5IAE52MWMVE7ICVMER76SBFCBPKOPP72PAJUA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.993796606 (59h25m41.184095195s) [noise=true , address=79.127.205.245:4225 , piece_id=I7JFEPX2Y4DPCQUOBU5TUKULWDKZ7NEPRZLEPYOAERT7BRMYLFWA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.993853704 (59h25m41.184152295s) [noise=true , address=77.90.31.19:4867 , piece_id=ZI2FNMGPOGZ64WLVV5GF2XYSU56PVUO3YCPHZVUB2SO5K4WN6JTQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.994216340 (59h25m41.184514922s) [noise=true , address=156.146.43.225:4434 , piece_id=FNPGLZZQ2YL2VXPS7BUYQW2DT4M7SN6X4DCVFG3HVH6PC6ZG7IVQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.994411232 (59h25m41.184709816s) [noise=true , address=156.146.43.225:4449 , piece_id=SEMN4XVA3KX445ENRUKO3YR67J5AWYR7YX7UIBYQCM3Y35R23T4A, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.994548199 (59h25m41.184846785s) [noise=true , address=79.127.220.101:4048 , piece_id=YPEXABOTQRMAJB6DXISMX2GIEQDPV63ZZ4YMF2WPAPH4CSLP4IEQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.994904683 (59h25m41.185203265s) [noise=true , address=77.90.31.15:4726 , piece_id=CBFIDL6QXZK7W3TAEFTU7W5SFBOILXH3J6K2DLRZPOK65F47PVVQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.994911722 (59h25m41.185210308s) [noise=true , address=45.91.116.5:4310 , piece_id=2PXTZYS2NFB7GFDRKKFW64GCANXPT62WK3TAB6IK44JKNO47RYNA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.995220245 (59h25m41.185518825s) [noise=true , address=79.127.205.234:4618 , piece_id=FWZ5R33KMNBPYXTEDU327QFOPA26UFZY4JBG2DF2E2BSEJ2UK3AA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.995325566 (59h25m41.185624154s) [noise=true , address=79.127.219.47:4061 , piece_id=EZBZCEEEARMWY3Y2B27I3ODZO3WXUQ4OD2SBTNBFEQ3QAYZU6JTA, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:09.995805465 (59h25m41.186104045s) [noise=true , address=79.127.205.246:4286 , piece_id=UPPM43A75WAW5JDNX7XHH2BI5RPOJU4ZRY3UIN7EKY6W7XQONYCQ, seg_pos={0 0}, upload=56993]: Uploading piece... Jul 26 23:01:10.394602149 (59h25m41.584900735s) [noise=true , address=79.127.219.51:4210 , piece_id=CKKBXLR4OTH2NCV355TZEMUMOT7FHELV4ZITVJV4M5UWK72JJWKA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:10.442664484 (59h25m41.632963066s) [noise=true , address=79.127.219.48:4090 , piece_id=AQTJGH6G2IYVSXJX3PGFWN3DPWKSWHKM5OFM7XD5SWRPUS2TD6VQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:10.533117292 (59h25m41.723415891s) [noise=true , address=79.127.219.38:4018 , piece_id=Q5UZCBJYPR6X67A5IAE52MWMVE7ICVMER76SBFCBPKOPP72PAJUA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: ecclient: upload failed (node:12waxNyWj1kBeWE3W1VtF94zw1berZJHeERXuBQYq7UMxERry2f, address:79.127.219.38:4018): protocol: storage node overloaded, request limit: 2000 Jul 26 23:01:10.570261899 (59h25m41.760560478s) [upload=56993]: Next returned nil segment Jul 26 23:01:10.570282169 (59h25m41.760580752s) [upload=56993]: Waiting for error group managing segments... Jul 26 23:01:10.927619792 (59h25m42.117918376s) [noise=true , address=79.127.220.101:4048 , piece_id=YPEXABOTQRMAJB6DXISMX2GIEQDPV63ZZ4YMF2WPAPH4CSLP4IEQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:11.080319500 (59h25m42.27061808s) [noise=true , address=156.146.43.225:4434 , piece_id=FNPGLZZQ2YL2VXPS7BUYQW2DT4M7SN6X4DCVFG3HVH6PC6ZG7IVQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:11.169558528 (59h25m42.359857115s) [noise=true , address=79.127.205.245:4252 , piece_id=4WRSOFZ2YH5ATAGFW4KSM3TGVPFFJUB3GCURDYLUSXCGJ2WHGEIA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:11.258443209 (59h25m42.448741783s) [noise=true , address=79.127.205.245:4225 , piece_id=I7JFEPX2Y4DPCQUOBU5TUKULWDKZ7NEPRZLEPYOAERT7BRMYLFWA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:11.276653660 (59h25m42.466952233s) [noise=true , address=79.127.205.234:4618 , piece_id=FWZ5R33KMNBPYXTEDU327QFOPA26UFZY4JBG2DF2E2BSEJ2UK3AA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:11.287821711 (59h25m42.47812028s) [noise=true , address=79.127.205.246:4286 , piece_id=UPPM43A75WAW5JDNX7XHH2BI5RPOJU4ZRY3UIN7EKY6W7XQONYCQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:11.614448918 (59h25m42.804747489s) [noise=true , address=45.91.116.14:4222 , piece_id=M62GI27NNODLKM5LVFG5D4CECC72IORZI23ODYJ4PU3GANMNI3JA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:12.617092285 (59h25m43.807390868s) [noise=true , address=156.146.43.225:4449 , piece_id=SEMN4XVA3KX445ENRUKO3YR67J5AWYR7YX7UIBYQCM3Y35R23T4A, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:12.685170284 (59h25m43.875468863s) [noise=true , address=79.127.219.47:4061 , piece_id=EZBZCEEEARMWY3Y2B27I3ODZO3WXUQ4OD2SBTNBFEQ3QAYZU6JTA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:17.133944232 (59h25m48.324242818s) [noise=true , address=79.127.219.38:4007 , piece_id=MU2WCMJ6NWIS5Y3SIXUBH3CNO3L4OHDYQ7UGGF7IGZ6SJAMEAS3Q, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:20.483064266 (59h25m51.673362831s) [noise=true , address=77.90.31.12:4617 , piece_id=7LPFNSFFYRFWJSC5VK357I56RVWCI6QESSPDLNEAUQOSGJKPXGPQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:21.892620878 (59h25m53.082919464s) [noise=true , address=77.90.31.19:4867 , piece_id=ZI2FNMGPOGZ64WLVV5GF2XYSU56PVUO3YCPHZVUB2SO5K4WN6JTQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:01:23.946316134 (59h25m55.136614713s) [noise=true , address=45.91.116.5:4310 , piece_id=2PXTZYS2NFB7GFDRKKFW64GCANXPT62WK3TAB6IK44JKNO47RYNA, seg_pos={0 0}, upload=56993]: Done uploading piece. err: <nil> Jul 26 23:02:09.500152257 (59h26m40.690450836s) [noise=true , address=77.90.31.15:4726 , piece_id=CBFIDL6QXZK7W3TAEFTU7W5SFBOILXH3J6K2DLRZPOK65F47PVVQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: write tcp 139.59.144.22:55762->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:09.500243458 (59h26m40.690542038s) [noise=true , address=77.90.31.11:4585 , piece_id=Y7YAKWFFFHW7HARAQDGOMKPZDQLXC72SD6FNO73YYOJGW4CTNGJQ, seg_pos={0 0}, upload=56993]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: write tcp 139.59.144.22:32820->77.90.31.11:4585: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:02:09.500297624 (59h26m40.690596208s) [seg_pos={0 0}, upload=56993]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:02:09.612617597 (59h26m40.802916183s) [upload=56993]: Done uploading segments... Jul 26 23:02:09.612650590 (59h26m40.802949171s) [upload=56993]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.115491769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.............................X..................X] duration: 1.582823722s FAIL (2025-07-26 23:01:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 1.582857196s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 824.138405ms FAIL (2025-07-26 23:01:52 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 824.167587ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................................X..] duration: 4.425615616s FAIL (2025-07-26 23:01:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 280->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:44.058547940 (59h24m15.24884652s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:59:44.058641678 (59h24m15.248940257s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:59:44.058646789 (59h24m15.248945359s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:59:44.224851647 (59h24m15.415150235s) [upload=56969]: Done uploading segments... Jul 26 22:59:44.224876852 (59h24m15.415175436s) [upload=56969]: Upload finished. err: <nil> Jul 26 23:00:00.341276866 (59h24m31.531575453s) [upload=56972]: Starting upload Jul 26 23:00:00.341377270 (59h24m31.531675865s) [upload=56972]: Done starting upload Jul 26 23:00:00.341505637 (59h24m31.531804221s) [upload=56972]: Uploading segments... Jul 26 23:00:00.341940351 (59h24m31.532238944s) [upload=56972]: Got next segment. Inline: false Jul 26 23:00:00.342010661 (59h24m31.532309252s) [upload=56972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:00.342017759 (59h24m31.532316348s) [upload=56972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:00.455411486 (59h24m31.64571009s) [seg_pos={0 0}, upload=56972]: Begin upload segment... Jul 26 23:00:00.455493514 (59h24m31.645792104s) [seg_pos={0 0}, upload=56972]: Done begin upload segment. Jul 26 23:00:00.455540024 (59h24m31.645838613s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.455561207 (59h24m31.645859795s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456074349 (59h24m31.646372943s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456243971 (59h24m31.64654256s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456605120 (59h24m31.646903743s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456973436 (59h24m31.647272026s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457050035 (59h24m31.647348625s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457398570 (59h24m31.64769716s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457455587 (59h24m31.647754178s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457779166 (59h24m31.648077758s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458115180 (59h24m31.648413769s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458194315 (59h24m31.648492905s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458592315 (59h24m31.648890905s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458651549 (59h24m31.648950139s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.459145371 (59h24m31.649443962s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.462726202 (59h24m31.653024793s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:01.079541782 (59h24m32.269840354s) [upload=56972]: Next returned nil segment Jul 26 23:00:01.079566166 (59h24m32.269864784s) [upload=56972]: Waiting for error group managing segments... Jul 26 23:00:01.210918643 (59h24m32.401217229s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.228724770 (59h24m32.419023355s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232110801 (59h24m32.422409369s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232305605 (59h24m32.422604172s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234390772 (59h24m32.424689363s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234614195 (59h24m32.424912769s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246294708 (59h24m32.43659329s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246456357 (59h24m32.43675493s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.247086738 (59h24m32.437385324s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.248804736 (59h24m32.43910332s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.302224542 (59h24m32.492523129s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:30.042884997 (59h25m1.233183578s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:48592->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043116623 (59h25m1.233415194s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043228739 (59h25m1.233527306s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043315635 (59h25m1.233614206s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043325410 (59h25m1.233623995s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:50064->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043373737 (59h25m1.23367233s) [seg_pos={0 0}, upload=56972]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:00:30.159169528 (59h25m1.349468101s) [upload=56972]: Done uploading segments... Jul 26 23:00:30.159218327 (59h25m1.34951691s) [upload=56972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 4.425730839s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.....X.......X..X...X.XX...X..X.XX......X.X.......X.......X.] duration: 14.346029246s FAIL (2025-07-26 23:01:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.3460433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.............................X..................] duration: 2.85856121s FAIL (2025-07-26 23:01:31 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.858590436s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [......X.......X..X...X.XX...X..X.XX......X.X.......X.......X] duration: 1m0.000730756s FAIL (2025-07-26 23:01:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000749597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXXX] duration: 1m0.108232075s FAIL (2025-07-26 23:01:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== oad=56976]: Got next segment. Inline: false Jul 26 23:00:09.516911637 (59h24m40.707210226s) [upload=56976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:09.516918328 (59h24m40.707216911s) [upload=56976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:09.716565275 (59h24m40.906863866s) [seg_pos={0 0}, upload=56976]: Begin upload segment... Jul 26 23:00:09.716655589 (59h24m40.906954175s) [seg_pos={0 0}, upload=56976]: Done begin upload segment. Jul 26 23:00:09.716704658 (59h24m40.907003253s) [noise=true , address=45.91.116.14:4194 , piece_id=CHTR75M4HONNIFY3JRIJ26JDVWWRBYZ36ZP3ZLRPVG5XOKVF7VQA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.717353013 (59h24m40.907651603s) [noise=true , address=156.146.43.225:4449 , piece_id=NRKZ7VIZDI77ROJRFOTN2HM74FI22PS7ZQPPYVF2F6NBASOY3Z6A, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.717819018 (59h24m40.908117606s) [noise=true , address=77.90.31.15:4720 , piece_id=VT7BNY2ILCMUM7DEAIG6OF72UT2WSBHFQGDJZ3YRVRSZPQKZZTFA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.718237161 (59h24m40.908535753s) [noise=true , address=77.90.31.11:4557 , piece_id=DBBBDTESFRNT3FWLITLOM4IJ66YX4ANR4WBZCWYKDE2GXBCCOIAA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.718613973 (59h24m40.908912562s) [noise=true , address=77.90.31.16:4743 , piece_id=GCPBLFXDAFJ7UFENOSF6GDBD7ZKYJO3KFJRZ247LOBSO5XPVLIPQ, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.718999044 (59h24m40.909297636s) [noise=true , address=79.127.205.234:4604 , piece_id=GAFURZO2XPKV7S4NUFTP4CZJYU2VK5RHARKZUH6PVS4V7P6ISTWQ, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.719431769 (59h24m40.909730358s) [noise=true , address=79.127.205.245:4245 , piece_id=RX3V7TKDMJYBYYQCDZ5XCUZPXN7TAW2NX4VP5FX4IIVIANHZOZ7Q, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.719799106 (59h24m40.910097695s) [noise=true , address=79.127.205.235:4668 , piece_id=BWNU2ZHKWFCOVXS4QGDTWFAIMAKOPTCVCLOIBNRT5NIO7XTREPYQ, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.720152643 (59h24m40.910451234s) [noise=true , address=79.127.205.235:4689 , piece_id=TBFFSLW52B3MKAAVDE26SMPQJCT34X5QLJYWBLORLSAMILRN2YWA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.720598122 (59h24m40.910896711s) [noise=true , address=79.127.205.246:4261 , piece_id=B54I6UIC4S3YEJDVMNS3J6O6RLNN6BPMXDI5FOZKX56FQQFVAGHA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.720980845 (59h24m40.911279439s) [noise=true , address=79.127.219.38:4018 , piece_id=NSY53WOGVT5S7PHCP5T64JIVI4QOFL6JKL4CTE3WTHWETJVKOK5Q, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.721339878 (59h24m40.91163847s) [noise=true , address=77.90.31.19:4867 , piece_id=STSKB6GRJFEWHLPQEVIGRJKA4W5S6OSGRCUAUKMPRWNANWFJMSNA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.721752157 (59h24m40.912050747s) [noise=true , address=79.127.219.38:4013 , piece_id=LGJCC7TKSTOQTJZAWQ3MXOJJAH2RD5ZPAT4VYEZCGVLWKUFH6E3A, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.722156469 (59h24m40.912455061s) [noise=true , address=45.91.116.9:4267 , piece_id=LDORXVKHBNSSDOZORNBDKUEYJXDRTCBJKJHC5STYFOBDEIOAVFSA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.722504352 (59h24m40.912802943s) [noise=true , address=77.90.31.21:4909 , piece_id=AD5HHEAZUZWAGVUUL6QTD7YESAUWLB3BAABDUVX535GWHPMWWKRQ, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.722915136 (59h24m40.913213728s) [noise=true , address=64.49.13.140:4515 , piece_id=VRDW3FAENLS4W3DTFINZ4GIXJWBOU2VRPTUWRIQAQGE5VX4XDISA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.723264401 (59h24m40.913562994s) [noise=true , address=79.127.219.47:4048 , piece_id=YQ2PUFP2NDXVCBBT75UXVQJQL4JJLKLOAKH2HLM6DV2SZK4JRQZA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:09.723616764 (59h24m40.913915359s) [noise=true , address=77.90.31.10:4530 , piece_id=WT2I7V2VAQNYD54PXJTXXO5CZ4DQW6YNOW5G7NTG6BZMEQEZJQAA, seg_pos={0 0}, upload=56976]: Uploading piece... Jul 26 23:00:10.307674436 (59h24m41.497973025s) [upload=56976]: Next returned nil segment Jul 26 23:00:10.307699031 (59h24m41.497997622s) [upload=56976]: Waiting for error group managing segments... Jul 26 23:00:10.514618716 (59h24m41.70491729s) [noise=true , address=45.91.116.9:4267 , piece_id=LDORXVKHBNSSDOZORNBDKUEYJXDRTCBJKJHC5STYFOBDEIOAVFSA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:10.990425728 (59h24m42.18072439s) [noise=true , address=64.49.13.140:4515 , piece_id=VRDW3FAENLS4W3DTFINZ4GIXJWBOU2VRPTUWRIQAQGE5VX4XDISA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:11.007644907 (59h24m42.197943478s) [noise=true , address=79.127.205.246:4261 , piece_id=B54I6UIC4S3YEJDVMNS3J6O6RLNN6BPMXDI5FOZKX56FQQFVAGHA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:11.022486663 (59h24m42.212785253s) [noise=true , address=79.127.205.245:4245 , piece_id=RX3V7TKDMJYBYYQCDZ5XCUZPXN7TAW2NX4VP5FX4IIVIANHZOZ7Q, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:11.030040637 (59h24m42.220339219s) [noise=true , address=79.127.205.234:4604 , piece_id=GAFURZO2XPKV7S4NUFTP4CZJYU2VK5RHARKZUH6PVS4V7P6ISTWQ, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:11.047174890 (59h24m42.237473464s) [noise=true , address=156.146.43.225:4449 , piece_id=NRKZ7VIZDI77ROJRFOTN2HM74FI22PS7ZQPPYVF2F6NBASOY3Z6A, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:11.085623871 (59h24m42.275922458s) [noise=true , address=79.127.205.235:4668 , piece_id=BWNU2ZHKWFCOVXS4QGDTWFAIMAKOPTCVCLOIBNRT5NIO7XTREPYQ, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:11.150382436 (59h24m42.34068101s) [noise=true , address=79.127.205.235:4689 , piece_id=TBFFSLW52B3MKAAVDE26SMPQJCT34X5QLJYWBLORLSAMILRN2YWA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:11.806242977 (59h24m42.996541545s) [noise=true , address=79.127.219.38:4013 , piece_id=LGJCC7TKSTOQTJZAWQ3MXOJJAH2RD5ZPAT4VYEZCGVLWKUFH6E3A, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:13.315662724 (59h24m44.505961303s) [noise=true , address=45.91.116.14:4194 , piece_id=CHTR75M4HONNIFY3JRIJ26JDVWWRBYZ36ZP3ZLRPVG5XOKVF7VQA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 26 23:00:13.871951760 (59h24m45.06225034s) [noise=true , address=79.127.219.38:4018 , piece_id=NSY53WOGVT5S7PHCP5T64JIVI4QOFL6JKL4CTE3WTHWETJVKOK5Q, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:18.230684431 (59h24m49.420983014s) [noise=true , address=79.127.219.47:4048 , piece_id=YQ2PUFP2NDXVCBBT75UXVQJQL4JJLKLOAKH2HLM6DV2SZK4JRQZA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:00:38.046854917 (59h25m9.237153495s) [noise=true , address=77.90.31.19:4867 , piece_id=STSKB6GRJFEWHLPQEVIGRJKA4W5S6OSGRCUAUKMPRWNANWFJMSNA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: <nil> Jul 26 23:01:09.241409825 (59h25m40.431708415s) [noise=true , address=77.90.31.10:4530 , piece_id=WT2I7V2VAQNYD54PXJTXXO5CZ4DQW6YNOW5G7NTG6BZMEQEZJQAA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: write tcp 139.59.144.22:54504->77.90.31.10:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:09.241484627 (59h25m40.431783193s) [noise=true , address=77.90.31.15:4720 , piece_id=VT7BNY2ILCMUM7DEAIG6OF72UT2WSBHFQGDJZ3YRVRSZPQKZZTFA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:09.241599843 (59h25m40.431898422s) [noise=true , address=77.90.31.16:4743 , piece_id=GCPBLFXDAFJ7UFENOSF6GDBD7ZKYJO3KFJRZ247LOBSO5XPVLIPQ, seg_pos={0 0}, upload=56976]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: rpc: dial tcp 77.90.31.16:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:09.241669113 (59h25m40.431967693s) [noise=true , address=77.90.31.11:4557 , piece_id=DBBBDTESFRNT3FWLITLOM4IJ66YX4ANR4WBZCWYKDE2GXBCCOIAA, seg_pos={0 0}, upload=56976]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: rpc: dial tcp 77.90.31.11:4557: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:09.241823532 (59h25m40.432122132s) [noise=true , address=77.90.31.21:4909 , piece_id=AD5HHEAZUZWAGVUUL6QTD7YESAUWLB3BAABDUVX535GWHPMWWKRQ, seg_pos={0 0}, upload=56976]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: read tcp 139.59.144.22:53736->77.90.31.21:4909: use of closed network connection; context deadline exceeded Jul 26 23:01:09.241884036 (59h25m40.432182627s) [seg_pos={0 0}, upload=56976]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:01:09.346320004 (59h25m40.536618578s) [upload=56976]: Done uploading segments... Jul 26 23:01:09.346354234 (59h25m40.536652813s) [upload=56976]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.108250789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................................X.] duration: 2.423341635s FAIL (2025-07-26 23:00:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 280->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:44.058547940 (59h24m15.24884652s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:59:44.058641678 (59h24m15.248940257s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:59:44.058646789 (59h24m15.248945359s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:59:44.224851647 (59h24m15.415150235s) [upload=56969]: Done uploading segments... Jul 26 22:59:44.224876852 (59h24m15.415175436s) [upload=56969]: Upload finished. err: <nil> Jul 26 23:00:00.341276866 (59h24m31.531575453s) [upload=56972]: Starting upload Jul 26 23:00:00.341377270 (59h24m31.531675865s) [upload=56972]: Done starting upload Jul 26 23:00:00.341505637 (59h24m31.531804221s) [upload=56972]: Uploading segments... Jul 26 23:00:00.341940351 (59h24m31.532238944s) [upload=56972]: Got next segment. Inline: false Jul 26 23:00:00.342010661 (59h24m31.532309252s) [upload=56972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:00.342017759 (59h24m31.532316348s) [upload=56972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:00.455411486 (59h24m31.64571009s) [seg_pos={0 0}, upload=56972]: Begin upload segment... Jul 26 23:00:00.455493514 (59h24m31.645792104s) [seg_pos={0 0}, upload=56972]: Done begin upload segment. Jul 26 23:00:00.455540024 (59h24m31.645838613s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.455561207 (59h24m31.645859795s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456074349 (59h24m31.646372943s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456243971 (59h24m31.64654256s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456605120 (59h24m31.646903743s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456973436 (59h24m31.647272026s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457050035 (59h24m31.647348625s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457398570 (59h24m31.64769716s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457455587 (59h24m31.647754178s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457779166 (59h24m31.648077758s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458115180 (59h24m31.648413769s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458194315 (59h24m31.648492905s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458592315 (59h24m31.648890905s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458651549 (59h24m31.648950139s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.459145371 (59h24m31.649443962s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.462726202 (59h24m31.653024793s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:01.079541782 (59h24m32.269840354s) [upload=56972]: Next returned nil segment Jul 26 23:00:01.079566166 (59h24m32.269864784s) [upload=56972]: Waiting for error group managing segments... Jul 26 23:00:01.210918643 (59h24m32.401217229s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.228724770 (59h24m32.419023355s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232110801 (59h24m32.422409369s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232305605 (59h24m32.422604172s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234390772 (59h24m32.424689363s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234614195 (59h24m32.424912769s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246294708 (59h24m32.43659329s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246456357 (59h24m32.43675493s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.247086738 (59h24m32.437385324s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.248804736 (59h24m32.43910332s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.302224542 (59h24m32.492523129s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:30.042884997 (59h25m1.233183578s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:48592->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043116623 (59h25m1.233415194s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043228739 (59h25m1.233527306s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043315635 (59h25m1.233614206s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043325410 (59h25m1.233623995s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:50064->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043373737 (59h25m1.23367233s) [seg_pos={0 0}, upload=56972]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:00:30.159169528 (59h25m1.349468101s) [upload=56972]: Done uploading segments... Jul 26 23:00:30.159218327 (59h25m1.34951691s) [upload=56972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.423375647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 816.021151ms FAIL (2025-07-26 23:00:37 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 816.053556ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................................X] duration: 30.118864153s FAIL (2025-07-26 23:00:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 280->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:44.058547940 (59h24m15.24884652s) [seg_pos={0 0}, upload=56969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:59:44.058641678 (59h24m15.248940257s) [upload=56969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:59:44.058646789 (59h24m15.248945359s) [upload=56969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:59:44.224851647 (59h24m15.415150235s) [upload=56969]: Done uploading segments... Jul 26 22:59:44.224876852 (59h24m15.415175436s) [upload=56969]: Upload finished. err: <nil> Jul 26 23:00:00.341276866 (59h24m31.531575453s) [upload=56972]: Starting upload Jul 26 23:00:00.341377270 (59h24m31.531675865s) [upload=56972]: Done starting upload Jul 26 23:00:00.341505637 (59h24m31.531804221s) [upload=56972]: Uploading segments... Jul 26 23:00:00.341940351 (59h24m31.532238944s) [upload=56972]: Got next segment. Inline: false Jul 26 23:00:00.342010661 (59h24m31.532309252s) [upload=56972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:00:00.342017759 (59h24m31.532316348s) [upload=56972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:00:00.455411486 (59h24m31.64571009s) [seg_pos={0 0}, upload=56972]: Begin upload segment... Jul 26 23:00:00.455493514 (59h24m31.645792104s) [seg_pos={0 0}, upload=56972]: Done begin upload segment. Jul 26 23:00:00.455540024 (59h24m31.645838613s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.455561207 (59h24m31.645859795s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456074349 (59h24m31.646372943s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456243971 (59h24m31.64654256s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456605120 (59h24m31.646903743s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.456973436 (59h24m31.647272026s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457050035 (59h24m31.647348625s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457398570 (59h24m31.64769716s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457455587 (59h24m31.647754178s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.457779166 (59h24m31.648077758s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458115180 (59h24m31.648413769s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458194315 (59h24m31.648492905s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458592315 (59h24m31.648890905s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.458651549 (59h24m31.648950139s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.459145371 (59h24m31.649443962s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:00.462726202 (59h24m31.653024793s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Uploading piece... Jul 26 23:00:01.079541782 (59h24m32.269840354s) [upload=56972]: Next returned nil segment Jul 26 23:00:01.079566166 (59h24m32.269864784s) [upload=56972]: Waiting for error group managing segments... Jul 26 23:00:01.210918643 (59h24m32.401217229s) [noise=true , address=45.91.116.12:4226 , piece_id=ZOJ5SOHVEXCAVZVF7OMD27Y6DGC5NTQ6KHVXKPAJC3OBU2FBIXOQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.228724770 (59h24m32.419023355s) [noise=true , address=45.91.116.11:4119 , piece_id=YG4CWZ7AXI63LDZYGATKBUNHYN7K6KUWP6D3TVHVZ6BZWI7CUZ4Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232110801 (59h24m32.422409369s) [noise=true , address=45.91.116.5:4299 , piece_id=NPKPUUP2IWRY73HRKRO5OCW5OFORR7YX5JYHGJTGUWG4SUQFE2TA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.232305605 (59h24m32.422604172s) [noise=true , address=45.91.116.12:4225 , piece_id=HVTSJDCOEKNPYAEP6FC4EUEW3VJXODSYCOB77KA2MYDNX3ML35PA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234390772 (59h24m32.424689363s) [noise=true , address=77.90.31.14:4662 , piece_id=M4UEGWQ3Q573A3W7QFY66V7XVBA6BULAJSLFDSASKATFGDMIWJ2A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.234614195 (59h24m32.424912769s) [noise=true , address=45.91.116.5:4297 , piece_id=XJGSWZ4RX3CQ3LB6JIPYYPCNOQIX24K7CRJ4DUXLSZ2SH74R5M2Q, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246294708 (59h24m32.43659329s) [noise=true , address=45.91.116.13:4154 , piece_id=H4MIPFF5OQ5AN4LEVKMIZBAGHXYUTYTCQ3UJFG5GBLWA2E33QS3A, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.246456357 (59h24m32.43675493s) [noise=true , address=45.91.116.7:4009 , piece_id=YOYTKIIWKKRS5W7MTN6ZA4ZYXNRCNVSTRBLZHOYO5S34MUDFOPLA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.247086738 (59h24m32.437385324s) [noise=true , address=45.91.116.13:4155 , piece_id=6JAU4O6DIWNNOBEYAW4O7LBIAHYD2SUSJPG3DWGUTN7GFJH6ECYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.248804736 (59h24m32.43910332s) [noise=true , address=45.91.116.8:4048 , piece_id=7I4KIRKCHCY3ORFS45APNHF5JCCQNQU7LYBUJ6ZDZZCUVAVICCRA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:01.302224542 (59h24m32.492523129s) [noise=true , address=45.91.116.10:4083 , piece_id=BOZWSVQ2VD4TC5F6N3R4NWD4Y3OFTV7IKKLYMFY23IKPUG35HSJQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: <nil> Jul 26 23:00:30.042884997 (59h25m1.233183578s) [noise=true , address=77.90.31.11:4554 , piece_id=2OP6YK5MGAV3PVWDATCD26O5IORL7G3B4RFIBRAUKBW63FVTE6WA, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 139.59.144.22:48592->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043116623 (59h25m1.233415194s) [noise=true , address=77.90.31.18:4806 , piece_id=OD3O5OIS4ZCVPUDJU55EARB3TKZSVQJYS5HR5ZUEVKZKNDKTPFPQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043228739 (59h25m1.233527306s) [noise=true , address=77.90.31.16:4733 , piece_id=5K5CRJ45TVLPKE6APTAETRE5T32VOAT5V5BUWP4WRHUV5JIHLYZQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043315635 (59h25m1.233614206s) [noise=true , address=77.90.31.21:4877 , piece_id=7OBZ2Q6DYTCCVQTZNLTHEXFZDWFJDHHH64CJJMNBNJZTUIAX3GHQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043325410 (59h25m1.233623995s) [noise=true , address=77.90.31.11:4553 , piece_id=PPQIED5COJYGR5N7MZDY3UL6BTW6OW3AAT4GG4D4CYUKY53BQAYQ, seg_pos={0 0}, upload=56972]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 139.59.144.22:50064->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:30.043373737 (59h25m1.23367233s) [seg_pos={0 0}, upload=56972]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:00:30.159169528 (59h25m1.349468101s) [upload=56972]: Done uploading segments... Jul 26 23:00:30.159218327 (59h25m1.34951691s) [upload=56972]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.118888825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.............................X.................] duration: 2.367581955s FAIL (2025-07-26 23:00:29 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.36761822s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 856.350198ms FAIL (2025-07-26 23:00:13 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 856.386965ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XXX] duration: 1m0.116939537s FAIL (2025-07-26 23:00:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ) [upload=56962]: Done starting upload Jul 26 22:59:09.249641359 (59h23m40.43993993s) [upload=56962]: Uploading segments... Jul 26 22:59:09.249897282 (59h23m40.440195852s) [upload=56962]: Got next segment. Inline: false Jul 26 22:59:09.249974416 (59h23m40.440272999s) [upload=56962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:59:09.249982882 (59h23m40.440281449s) [upload=56962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:59:09.493473280 (59h23m40.683771862s) [seg_pos={0 0}, upload=56962]: Begin upload segment... Jul 26 22:59:09.493569573 (59h23m40.683868161s) [noise=true , address=77.90.31.11:4580 , piece_id=3ZSEJT5FIODQIEKZLPRTRR4K2AARFCCY4BR2VCGAIMMVZRKT7MTA, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.494056892 (59h23m40.684355808s) [noise=true , address=77.90.31.15:4732 , piece_id=6F5IL3TGYTGATGBKQBWQHGLIFGSLS5NHL6AJJ2SF6ET25ZLDLSLA, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.494294600 (59h23m40.684593188s) [seg_pos={0 0}, upload=56962]: Done begin upload segment. Jul 26 22:59:09.494352284 (59h23m40.684650869s) [noise=true , address=79.127.219.38:4010 , piece_id=2JOVQJGNBFEZH2Y7PCGZ5KGVCDWE3B4TYWLID7GV6OYQEWRPRDXQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.494436686 (59h23m40.684735269s) [noise=true , address=77.90.31.13:4629 , piece_id=GS3RT3MSBJ2HONO5ZMDBHBMG5PCIFPGAZNQVNRSKWAEN2I5CGJPQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.494731974 (59h23m40.685030554s) [noise=true , address=45.91.116.11:4121 , piece_id=K7SSJJDQSMVCKM5VIONFOCADOFKLD5MM3NBFIISZEHYUXI2MGPQQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.494841400 (59h23m40.68513999s) [noise=true , address=79.127.205.234:4612 , piece_id=4ARUUEX5NAGCGWMVTLIXH5Q3PZHBL5GJZC6M4SS55TZ27A5ACZMQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.495060955 (59h23m40.685359792s) [noise=true , address=79.127.219.38:4005 , piece_id=XS7DT6OS2NL2CF4Y3ZIW5UZPP76E5U643OTWX3FYSRYYWBQVR2FQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.495332704 (59h23m40.685631287s) [noise=true , address=79.127.201.215:4344 , piece_id=BAI7DFZHNNQNIJQ6IL2ZDQZ4MIPOKYROBVHM4WI5SQNXYGV4KFNQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.496257021 (59h23m40.686555607s) [noise=true , address=79.127.205.234:4603 , piece_id=UUJWRDHOZLUE4BCPJUXZHK66JGJZBGH7DDZ6RAYAQUCTGG2DCZXQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.496858792 (59h23m40.687157375s) [noise=true , address=79.127.205.233:4639 , piece_id=5GAH2RNZYA66RCKERSBV3R5TGDNNHE6GTQALJTT3LWRRWY73GG6Q, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.497277398 (59h23m40.687575988s) [noise=true , address=79.127.205.245:4247 , piece_id=GT443BVHLQASHFSEFNO2VKHRGL5AMKJ4OSRZNNACVU5WDKHSLTUA, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.497715854 (59h23m40.688014433s) [noise=true , address=156.146.43.225:4434 , piece_id=VH6WZ3CJ66IO7JVGVMKBPPI2CRKEO7VEFYI7Z5Z2ZLVJEYDPBCXQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.498242488 (59h23m40.688541073s) [noise=true , address=64.49.13.170:4718 , piece_id=FWYEDUEI4K4OYKUOJAAFGTQQ4U3C6SOTXZSDTJ35D7QZOHHW7JXQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.498643027 (59h23m40.68894161s) [noise=true , address=79.127.219.51:4212 , piece_id=VPU3IGO5EUO73EQ3EGL2O7Q5OVW7GPK2CWNKLSK7JX55VUZHCT3A, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.502187218 (59h23m40.692485803s) [noise=true , address=156.146.43.225:4449 , piece_id=WDFD5BQTRQJIDLJA3GJ3HZYOVQ4RNX5DNT666JE4G47O7KEPUZMQ, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.502526609 (59h23m40.692825194s) [noise=true , address=45.91.116.9:4277 , piece_id=5QNU74EJT5CPDCIR6IJBIOKFPHTLY37XQQ3TARPDACWZIAUKMOZA, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.502788014 (59h23m40.693086593s) [noise=true , address=79.127.205.246:4282 , piece_id=4SZN2ZPWKDNMCZTFIVPX2WJOZJHQI6LBPT6DFG5IMAAF4CNAG6EA, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.503019255 (59h23m40.693317878s) [noise=true , address=77.90.31.16:4767 , piece_id=PZOVCZCQPY5ZV4WRFYGP2DCVSP4BPBTM4X42GHUPSA6KI5E56D7A, seg_pos={0 0}, upload=56962]: Uploading piece... Jul 26 22:59:09.986906310 (59h23m41.177204915s) [noise=true , address=45.91.116.9:4277 , piece_id=5QNU74EJT5CPDCIR6IJBIOKFPHTLY37XQQ3TARPDACWZIAUKMOZA, seg_pos={0 0}, upload=56962]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:10.075180440 (59h23m41.265479021s) [upload=56962]: Next returned nil segment Jul 26 22:59:10.075199002 (59h23m41.265497572s) [upload=56962]: Waiting for error group managing segments... Jul 26 22:59:10.494237370 (59h23m41.684535953s) [noise=true , address=79.127.219.38:4005 , piece_id=XS7DT6OS2NL2CF4Y3ZIW5UZPP76E5U643OTWX3FYSRYYWBQVR2FQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:10.777583304 (59h23m41.967881878s) [noise=true , address=156.146.43.225:4449 , piece_id=WDFD5BQTRQJIDLJA3GJ3HZYOVQ4RNX5DNT666JE4G47O7KEPUZMQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:10.801202982 (59h23m41.991501547s) [noise=true , address=79.127.205.234:4603 , piece_id=UUJWRDHOZLUE4BCPJUXZHK66JGJZBGH7DDZ6RAYAQUCTGG2DCZXQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:10.902495360 (59h23m42.092793933s) [noise=true , address=79.127.205.234:4612 , piece_id=4ARUUEX5NAGCGWMVTLIXH5Q3PZHBL5GJZC6M4SS55TZ27A5ACZMQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:11.141984301 (59h23m42.332282866s) [noise=true , address=79.127.205.233:4639 , piece_id=5GAH2RNZYA66RCKERSBV3R5TGDNNHE6GTQALJTT3LWRRWY73GG6Q, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:13.715198952 (59h23m44.905497523s) [noise=true , address=79.127.219.51:4212 , piece_id=VPU3IGO5EUO73EQ3EGL2O7Q5OVW7GPK2CWNKLSK7JX55VUZHCT3A, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:14.151748526 (59h23m45.342047116s) [noise=true , address=79.127.219.38:4010 , piece_id=2JOVQJGNBFEZH2Y7PCGZ5KGVCDWE3B4TYWLID7GV6OYQEWRPRDXQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:14.531372001 (59h23m45.72167059s) [noise=true , address=156.146.43.225:4434 , piece_id=VH6WZ3CJ66IO7JVGVMKBPPI2CRKEO7VEFYI7Z5Z2ZLVJEYDPBCXQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:16.583223958 (59h23m47.773522539s) [noise=true , address=64.49.13.170:4718 , piece_id=FWYEDUEI4K4OYKUOJAAFGTQQ4U3C6SOTXZSDTJ35D7QZOHHW7JXQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:18.095932986 (59h23m49.286231556s) [noise=true , address=79.127.205.246:4282 , piece_id=4SZN2ZPWKDNMCZTFIVPX2WJOZJHQI6LBPT6DFG5IMAAF4CNAG6EA, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:18.761791320 (59h23m49.952089889s) [noise=true , address=45.91.116.11:4121 , piece_id=K7SSJJDQSMVCKM5VIONFOCADOFKLD5MM3NBFIISZEHYUXI2MGPQQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 22:59:27.512963907 (59h23m58.70326249s) [noise=true , address=79.127.205.245:4247 , piece_id=GT443BVHLQASHFSEFNO2VKHRGL5AMKJ4OSRZNNACVU5WDKHSLTUA, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 22:59:35.502523200 (59h24m6.692821773s) [noise=true , address=79.127.201.215:4344 , piece_id=BAI7DFZHNNQNIJQ6IL2ZDQZ4MIPOKYROBVHM4WI5SQNXYGV4KFNQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: <nil> Jul 26 23:00:08.981976879 (59h24m40.172275454s) [noise=true , address=77.90.31.16:4767 , piece_id=PZOVCZCQPY5ZV4WRFYGP2DCVSP4BPBTM4X42GHUPSA6KI5E56D7A, seg_pos={0 0}, upload=56962]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: rpc: dial tcp 77.90.31.16:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:08.981995293 (59h24m40.172293872s) [noise=true , address=77.90.31.11:4580 , piece_id=3ZSEJT5FIODQIEKZLPRTRR4K2AARFCCY4BR2VCGAIMMVZRKT7MTA, seg_pos={0 0}, upload=56962]: Done uploading piece. err: ecclient: upload failed (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, address:77.90.31.11:4580): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:00:08.982203008 (59h24m40.172501588s) [noise=true , address=77.90.31.15:4732 , piece_id=6F5IL3TGYTGATGBKQBWQHGLIFGSLS5NHL6AJJ2SF6ET25ZLDLSLA, seg_pos={0 0}, upload=56962]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:00:08.982234926 (59h24m40.172533495s) [noise=true , address=77.90.31.13:4629 , piece_id=GS3RT3MSBJ2HONO5ZMDBHBMG5PCIFPGAZNQVNRSKWAEN2I5CGJPQ, seg_pos={0 0}, upload=56962]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: write tcp 139.59.144.22:37818->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:08.982260116 (59h24m40.172558695s) [seg_pos={0 0}, upload=56962]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:00:09.096103674 (59h24m40.286402247s) [upload=56962]: Done uploading segments... Jul 26 23:00:09.096135659 (59h24m40.286434241s) [upload=56962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.11698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............................X................] duration: 2.324658783s FAIL (2025-07-26 23:00:09 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.324696126s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......X.......X..X...X.XX...X..X.XX......X.X.......X.......] duration: 28.170705371s FAIL (2025-07-26 22:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.170739938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.830242445s FAIL (2025-07-26 22:59:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 3.830272687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.............................X...............] duration: 2.430167216s FAIL (2025-07-26 22:59:09 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.43020417s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.XX] duration: 1m0.120789881s FAIL (2025-07-26 22:59:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ugh pieces (needed at least 14 but got 12) Jul 26 22:58:09.000077314 (59h22m40.190375908s) [upload=56946]: Starting upload Jul 26 22:58:09.000138955 (59h22m40.190437544s) [upload=56946]: Done starting upload Jul 26 22:58:09.000483991 (59h22m40.190782571s) [upload=56946]: Uploading segments... Jul 26 22:58:09.000655434 (59h22m40.190954036s) [upload=56946]: Got next segment. Inline: false Jul 26 22:58:09.000706317 (59h22m40.1910049s) [upload=56946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:58:09.000713090 (59h22m40.19101167s) [upload=56946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:58:09.211115586 (59h22m40.401414172s) [seg_pos={0 0}, upload=56946]: Begin upload segment... Jul 26 22:58:09.211197631 (59h22m40.401496197s) [seg_pos={0 0}, upload=56946]: Done begin upload segment. Jul 26 22:58:09.211243075 (59h22m40.401541667s) [noise=true , address=77.90.31.10:4539 , piece_id=C6YKOQ3MMZMBD54PW7U52QB3STWLPBP2JUZHM26WSSG2DEC6UL3Q, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.211642025 (59h22m40.401940625s) [noise=true , address=156.146.43.225:4449 , piece_id=5K2DLGVIQOGZ42BBQYXMCCVHERUOXOKJCP3GX5JIJAB4EXN2LOPA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.212191298 (59h22m40.402489874s) [noise=true , address=79.127.205.235:4678 , piece_id=MFY6LJCHH522FL5WJNVLIPREAAOUIKKCGCJXFCIGZLUEWCTHJQMA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.212873791 (59h22m40.403172382s) [noise=true , address=79.127.205.234:4610 , piece_id=PTRKCBDH3L5OGUNQYXXHMUQRFF6JQFH7RUV3D2ONGSTNDWEHTFEA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.213679754 (59h22m40.403978336s) [noise=true , address=79.127.205.246:4277 , piece_id=CO2OBLN2DTCTDGCUCERGBWJEFARJMUJORQQOQLTNX73YSA637KNQ, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.213820405 (59h22m40.404119018s) [noise=true , address=79.127.219.48:4090 , piece_id=K3WWZ23FUPQ3TPANLQ43BEVTPAXWDYMHAK7MJ3M5ZVG2PCFM35BQ, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.214608948 (59h22m40.404907529s) [noise=true , address=64.49.13.138:4483 , piece_id=7KTONWPRHCWVJ2VATL3MPWB7NYKHKY6DXVOVYTYDI6I3CPXNBOHQ, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.214665228 (59h22m40.40496381s) [noise=true , address=79.127.220.101:4048 , piece_id=SQQCK6OFEVPHXVR3OG4NBFVGPDFV6QOB67VSSGESXREVF6HW26EQ, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.215235061 (59h22m40.405533651s) [noise=true , address=77.90.31.11:4565 , piece_id=4NZNSMOU6TM5M5AA7VSPY4LPFN7M2IIUWCG7GWY5TCL2RBEXX2VA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.215488407 (59h22m40.405786996s) [noise=true , address=64.49.13.139:4478 , piece_id=5NJYYDGOBBPYLGV5XBSEEB2MXCT3UXPB4HLUQRHKDBGZIXEXYUHA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.215818292 (59h22m40.406116874s) [noise=true , address=45.91.116.14:4206 , piece_id=NSWIWSIEA23SSK7UPCWIPKI4NS47ZBB56WGZ3IEEEP4ZHAR3DE5Q, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.216056490 (59h22m40.406355071s) [noise=true , address=45.91.116.9:4277 , piece_id=2Y7CRNFJEWP64ZGA3SMVZND3HA62GFCRX4RR4WIQ6KZGQMKWSTZA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.216268112 (59h22m40.406566694s) [noise=true , address=79.127.201.215:4341 , piece_id=KFQNRLXACIBFEH7V6ZLVZ6XTURUGLLJQZEG7DNVXKUIE7JGWUEZQ, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.216533936 (59h22m40.406832525s) [noise=true , address=79.127.205.245:4249 , piece_id=F6MEKCFRTW36G2GJ7SB3PYIH3R6TDEXONMIVRDJ7VNP3PV5OOXTA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.216802374 (59h22m40.407100978s) [noise=true , address=79.127.219.47:4048 , piece_id=A2SQCVQL56Z2PWDUM64DZCEJJ5BVO6E7WLUTXZTR5TSSHX3RNCHA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.217061944 (59h22m40.407360528s) [noise=true , address=79.127.205.233:4652 , piece_id=VLQXG23HTXPC252E4LXRDPVWBITKENEDK2UVBLUQQBK34B4GRVOQ, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.217732351 (59h22m40.408030938s) [noise=true , address=77.90.31.13:4643 , piece_id=RHX4JOSDA7MGXXRTKPFBSS2BBU7VLBU6TC5OGBST46622X2BLLPQ, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.218056043 (59h22m40.408354628s) [noise=true , address=77.90.31.16:4746 , piece_id=DOVCRGPGJMJ63LJQIKB3GKRXBSZGF6Q3CETEVJZMQJITCZLOIYQA, seg_pos={0 0}, upload=56946]: Uploading piece... Jul 26 22:58:09.616251414 (59h22m40.806549984s) [noise=true , address=77.90.31.10:4539 , piece_id=C6YKOQ3MMZMBD54PW7U52QB3STWLPBP2JUZHM26WSSG2DEC6UL3Q, seg_pos={0 0}, upload=56946]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:09.640655867 (59h22m40.830954453s) [noise=true , address=79.127.219.47:4048 , piece_id=A2SQCVQL56Z2PWDUM64DZCEJJ5BVO6E7WLUTXZTR5TSSHX3RNCHA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:09.751687832 (59h22m40.941986412s) [noise=true , address=79.127.201.215:4341 , piece_id=KFQNRLXACIBFEH7V6ZLVZ6XTURUGLLJQZEG7DNVXKUIE7JGWUEZQ, seg_pos={0 0}, upload=56946]: Done uploading piece. err: ecclient: upload failed (node:12Pvou3YLLA7MA8WAfqjWLRwgwDXWPWZKGGRWYAVBkKCxF5mciG, address:79.127.201.215:4341): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:09.808198216 (59h22m40.9984968s) [upload=56946]: Next returned nil segment Jul 26 22:58:09.808218816 (59h22m40.998517398s) [upload=56946]: Waiting for error group managing segments... Jul 26 22:58:09.982105659 (59h22m41.172404249s) [noise=true , address=77.90.31.11:4565 , piece_id=4NZNSMOU6TM5M5AA7VSPY4LPFN7M2IIUWCG7GWY5TCL2RBEXX2VA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:10.103838707 (59h22m41.294137276s) [noise=true , address=79.127.220.101:4048 , piece_id=SQQCK6OFEVPHXVR3OG4NBFVGPDFV6QOB67VSSGESXREVF6HW26EQ, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:10.430549059 (59h22m41.62084763s) [noise=true , address=79.127.219.48:4090 , piece_id=K3WWZ23FUPQ3TPANLQ43BEVTPAXWDYMHAK7MJ3M5ZVG2PCFM35BQ, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:10.477470197 (59h22m41.667768766s) [noise=true , address=79.127.205.234:4610 , piece_id=PTRKCBDH3L5OGUNQYXXHMUQRFF6JQFH7RUV3D2ONGSTNDWEHTFEA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:10.503424264 (59h22m41.693722838s) [noise=true , address=79.127.205.235:4678 , piece_id=MFY6LJCHH522FL5WJNVLIPREAAOUIKKCGCJXFCIGZLUEWCTHJQMA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:10.602024146 (59h22m41.792322718s) [noise=true , address=79.127.205.245:4249 , piece_id=F6MEKCFRTW36G2GJ7SB3PYIH3R6TDEXONMIVRDJ7VNP3PV5OOXTA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:10.725870908 (59h22m41.916169489s) [noise=true , address=79.127.205.233:4652 , piece_id=VLQXG23HTXPC252E4LXRDPVWBITKENEDK2UVBLUQQBK34B4GRVOQ, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:10.748110290 (59h22m41.938408855s) [noise=true , address=79.127.205.246:4277 , piece_id=CO2OBLN2DTCTDGCUCERGBWJEFARJMUJORQQOQLTNX73YSA637KNQ, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:11.286945089 (59h22m42.477243654s) [noise=true , address=45.91.116.14:4206 , piece_id=NSWIWSIEA23SSK7UPCWIPKI4NS47ZBB56WGZ3IEEEP4ZHAR3DE5Q, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:13.154661916 (59h22m44.344960482s) [noise=true , address=156.146.43.225:4449 , piece_id=5K2DLGVIQOGZ42BBQYXMCCVHERUOXOKJCP3GX5JIJAB4EXN2LOPA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:14.854444714 (59h22m46.044743284s) [noise=true , address=64.49.13.138:4483 , piece_id=7KTONWPRHCWVJ2VATL3MPWB7NYKHKY6DXVOVYTYDI6I3CPXNBOHQ, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:18.814526912 (59h22m50.004825486s) [noise=true , address=64.49.13.139:4478 , piece_id=5NJYYDGOBBPYLGV5XBSEEB2MXCT3UXPB4HLUQRHKDBGZIXEXYUHA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:58:19.306945616 (59h22m50.497244187s) [noise=true , address=45.91.116.9:4277 , piece_id=2Y7CRNFJEWP64ZGA3SMVZND3HA62GFCRX4RR4WIQ6KZGQMKWSTZA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: <nil> Jul 26 22:59:08.719013768 (59h23m39.909312341s) [noise=true , address=77.90.31.16:4746 , piece_id=DOVCRGPGJMJ63LJQIKB3GKRXBSZGF6Q3CETEVJZMQJITCZLOIYQA, seg_pos={0 0}, upload=56946]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: write tcp 139.59.144.22:55390->77.90.31.16:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:08.719051147 (59h23m39.909349726s) [noise=true , address=77.90.31.13:4643 , piece_id=RHX4JOSDA7MGXXRTKPFBSS2BBU7VLBU6TC5OGBST46622X2BLLPQ, seg_pos={0 0}, upload=56946]: Done uploading piece. err: ecclient: upload failed (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA, address:77.90.31.13:4643): protocol: rpc: dial tcp 77.90.31.13:4643: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:08.719097845 (59h23m39.909396424s) [seg_pos={0 0}, upload=56946]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:59:08.837178060 (59h23m40.027476632s) [upload=56946]: Done uploading segments... Jul 26 22:59:08.837207426 (59h23m40.027506005s) [upload=56946]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.120815062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 835.18242ms FAIL (2025-07-26 22:59:06 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 835.216484ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.429091298s FAIL (2025-07-26 22:58:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.429132534s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........X.......X..X...X.XX...X..X.XX......X.X.......X......] duration: 10.369284475s FAIL (2025-07-26 22:58:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.369296809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.........X.......X..X...X.XX...X..X.XX......X.X.......X.....] duration: 57.977868321s FAIL (2025-07-26 22:58:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.977892173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.............................X..............] duration: 2.385037765s FAIL (2025-07-26 22:58:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.385067708s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 838.803642ms FAIL (2025-07-26 22:58:26 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 838.832529ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.253953313s FAIL (2025-07-26 22:58:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.25398902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [.X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.X] duration: 1m0.111520811s FAIL (2025-07-26 22:58:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ad=56912]: Upload finished. err: <nil> Jul 26 22:57:08.738019861 (59h21m39.928318441s) [upload=56928]: Starting upload Jul 26 22:57:08.738086949 (59h21m39.92838552s) [upload=56928]: Done starting upload Jul 26 22:57:08.738157188 (59h21m39.928455759s) [upload=56928]: Uploading segments... Jul 26 22:57:08.738384642 (59h21m39.928683221s) [upload=56928]: Got next segment. Inline: false Jul 26 22:57:08.738442946 (59h21m39.928741527s) [upload=56928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:57:08.738465330 (59h21m39.928763911s) [upload=56928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:57:08.995512821 (59h21m40.1858114s) [seg_pos={0 0}, upload=56928]: Begin upload segment... Jul 26 22:57:08.995574197 (59h21m40.185872776s) [seg_pos={0 0}, upload=56928]: Done begin upload segment. Jul 26 22:57:08.995620465 (59h21m40.185919047s) [noise=true , address=77.90.31.15:4712 , piece_id=H7NOOVM3XSQ6ESQ4BUXWHYKFGLRKYHIJPBC5PUKBXJ6ZLF6VENVA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.995655756 (59h21m40.185954347s) [noise=true , address=77.90.31.10:4523 , piece_id=UODIK7X26N2USCEDN2WHP5PRJEP7TIKVDMNYF64GFKFTKED5I46Q, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.996064692 (59h21m40.186363274s) [noise=true , address=79.127.219.51:4210 , piece_id=SMBFVLL2UVZVQWPB5AF6TSVOF5RSVRY5C7G3TJGVHMKT7G4HH3EQ, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.996131629 (59h21m40.186430217s) [noise=true , address=45.91.116.14:4206 , piece_id=DSWWENXCOQDZHD7Z26EGUXQGVRP2WK67FFZ2AF3TVQQVJFWEIGWA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.996632623 (59h21m40.186931207s) [noise=true , address=79.127.219.38:4009 , piece_id=3AS7JUMUBVDFA5YRTFXD7QB6GVDPVJ42HQ4XDTFMKI47RC5NHC4A, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.996722399 (59h21m40.187020988s) [noise=true , address=79.127.205.245:4236 , piece_id=OU52TBDP4IAODVU5BYDUCXDSPZTQV3V53E2G77SPQ2SYS7UYVHBQ, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.997003321 (59h21m40.187301906s) [noise=true , address=64.49.13.138:4483 , piece_id=XZIEJOXUQHJM5TPVWSAQ73HOKO67YK64XYRF375J2NISZZM4LICA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.997141037 (59h21m40.187439623s) [noise=true , address=64.49.13.139:4478 , piece_id=3W5DNWZC4SXKLEE7HKOC2P47MRW3EJCEAXXXZ2KX6LCBQPSTNZRA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.997389669 (59h21m40.187688258s) [noise=true , address=79.127.205.233:4633 , piece_id=JSL4H52VJSYAB7IRNHODIXELVP6QVMGSY5O4WCSTQYE73U7ZRJHA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.997525302 (59h21m40.187823887s) [noise=true , address=79.127.219.47:4048 , piece_id=VZUWII3RTRXSAK6W3GPWRXGN43QNKWIS33DE5KFQSETW5O5L5TJA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.997753351 (59h21m40.188051933s) [noise=true , address=79.127.205.234:4612 , piece_id=RG6HEJTE5BZJT7VOH4C76ZROK7RTHVKEDRADKGAQJIANXCXBSWUQ, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.997905785 (59h21m40.18820437s) [noise=true , address=77.90.31.13:4644 , piece_id=KSXILDQVRQD4364BUPPY7SX4YNUHBZAPVPIRAHMUXYQVWE7IHUUQ, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.998101112 (59h21m40.188399697s) [noise=true , address=77.90.31.21:4880 , piece_id=CSLEAOKZY2QI4ZLM7ZCE2C526WDBFPUZAP25RKSIUUVY2NDSW5DQ, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.998343872 (59h21m40.188642456s) [noise=true , address=79.127.220.101:4048 , piece_id=2V35HI5QH3WLSONADGOVOBQPXS72O65TJFR43YYHLP6HM2U645SA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.998408957 (59h21m40.188707546s) [noise=true , address=79.127.205.245:4251 , piece_id=4FFT57HHL6QFY46JZL274DHMZNFKLWBZQBTAAP5C7SX53YILUZ5A, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.998868367 (59h21m40.189166949s) [noise=true , address=156.146.43.225:4434 , piece_id=VJCYSXKE44QZYDYETBACFXVEXGQT3XLZPJNKUIHBP3XG2XRHLASQ, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:08.999214147 (59h21m40.18951273s) [noise=true , address=79.127.205.235:4691 , piece_id=2FU5OQLG3VT24NBGG7WI4MVD73MOLEGPHRS565JGDY6S5C7ZXEQA, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:09.002589595 (59h21m40.192888183s) [noise=true , address=79.127.201.215:4341 , piece_id=EQTRWPRDP6CIYUJG6ZIX3V2HJT3LZUIOT2IBLXZN63JAEVZ4NU6Q, seg_pos={0 0}, upload=56928]: Uploading piece... Jul 26 22:57:09.408071639 (59h21m40.598370228s) [noise=true , address=79.127.219.47:4048 , piece_id=VZUWII3RTRXSAK6W3GPWRXGN43QNKWIS33DE5KFQSETW5O5L5TJA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:09.432905507 (59h21m40.623204087s) [noise=true , address=77.90.31.10:4523 , piece_id=UODIK7X26N2USCEDN2WHP5PRJEP7TIKVDMNYF64GFKFTKED5I46Q, seg_pos={0 0}, upload=56928]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:09.554452294 (59h21m40.744750876s) [noise=true , address=156.146.43.225:4434 , piece_id=VJCYSXKE44QZYDYETBACFXVEXGQT3XLZPJNKUIHBP3XG2XRHLASQ, seg_pos={0 0}, upload=56928]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:09.562045111 (59h21m40.752343684s) [upload=56928]: Next returned nil segment Jul 26 22:57:09.562066101 (59h21m40.752364686s) [upload=56928]: Waiting for error group managing segments... Jul 26 22:57:09.699901674 (59h21m40.890200254s) [noise=true , address=45.91.116.14:4206 , piece_id=DSWWENXCOQDZHD7Z26EGUXQGVRP2WK67FFZ2AF3TVQQVJFWEIGWA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:09.769598461 (59h21m40.959897034s) [noise=true , address=77.90.31.21:4880 , piece_id=CSLEAOKZY2QI4ZLM7ZCE2C526WDBFPUZAP25RKSIUUVY2NDSW5DQ, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:09.808434865 (59h21m40.998733434s) [noise=true , address=79.127.219.38:4009 , piece_id=3AS7JUMUBVDFA5YRTFXD7QB6GVDPVJ42HQ4XDTFMKI47RC5NHC4A, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:10.130087344 (59h21m41.320385923s) [noise=true , address=64.49.13.139:4478 , piece_id=3W5DNWZC4SXKLEE7HKOC2P47MRW3EJCEAXXXZ2KX6LCBQPSTNZRA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:10.156128110 (59h21m41.34642669s) [noise=true , address=64.49.13.138:4483 , piece_id=XZIEJOXUQHJM5TPVWSAQ73HOKO67YK64XYRF375J2NISZZM4LICA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:10.227258312 (59h21m41.417556885s) [noise=true , address=79.127.205.235:4691 , piece_id=2FU5OQLG3VT24NBGG7WI4MVD73MOLEGPHRS565JGDY6S5C7ZXEQA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:10.268395811 (59h21m41.458694411s) [noise=true , address=79.127.205.233:4633 , piece_id=JSL4H52VJSYAB7IRNHODIXELVP6QVMGSY5O4WCSTQYE73U7ZRJHA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:10.272857959 (59h21m41.463156549s) [noise=true , address=79.127.205.234:4612 , piece_id=RG6HEJTE5BZJT7VOH4C76ZROK7RTHVKEDRADKGAQJIANXCXBSWUQ, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:10.273812180 (59h21m41.464110762s) [noise=true , address=79.127.205.245:4236 , piece_id=OU52TBDP4IAODVU5BYDUCXDSPZTQV3V53E2G77SPQ2SYS7UYVHBQ, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:11.254716352 (59h21m42.445014921s) [noise=true , address=79.127.201.215:4341 , piece_id=EQTRWPRDP6CIYUJG6ZIX3V2HJT3LZUIOT2IBLXZN63JAEVZ4NU6Q, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:12.960922852 (59h21m44.151221438s) [noise=true , address=79.127.219.51:4210 , piece_id=SMBFVLL2UVZVQWPB5AF6TSVOF5RSVRY5C7G3TJGVHMKT7G4HH3EQ, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:13.116380660 (59h21m44.306679242s) [noise=true , address=79.127.220.101:4048 , piece_id=2V35HI5QH3WLSONADGOVOBQPXS72O65TJFR43YYHLP6HM2U645SA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:15.288341834 (59h21m46.478640404s) [noise=true , address=79.127.205.245:4251 , piece_id=4FFT57HHL6QFY46JZL274DHMZNFKLWBZQBTAAP5C7SX53YILUZ5A, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:57:19.900103186 (59h21m51.090401775s) [noise=true , address=77.90.31.15:4712 , piece_id=H7NOOVM3XSQ6ESQ4BUXWHYKFGLRKYHIJPBC5PUKBXJ6ZLF6VENVA, seg_pos={0 0}, upload=56928]: Done uploading piece. err: <nil> Jul 26 22:58:08.459375750 (59h22m39.649674319s) [noise=true , address=77.90.31.13:4644 , piece_id=KSXILDQVRQD4364BUPPY7SX4YNUHBZAPVPIRAHMUXYQVWE7IHUUQ, seg_pos={0 0}, upload=56928]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:58:08.459409966 (59h22m39.649708548s) [seg_pos={0 0}, upload=56928]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:58:08.568908617 (59h22m39.759207209s) [upload=56928]: Done uploading segments... Jul 26 22:58:08.568940640 (59h22m39.759239227s) [upload=56928]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.111551667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 850.802304ms FAIL (2025-07-26 22:57:42 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 850.899778ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.326552936s FAIL (2025-07-26 22:57:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.32663716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.............................X.............] duration: 2.239048553s FAIL (2025-07-26 22:57:16 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 2.239078437s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..........X.......X..X...X.XX...X..X.XX......X.X.......X....] duration: 11.053657267s FAIL (2025-07-26 22:57:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.053686319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 841.771025ms FAIL (2025-07-26 22:56:45 UTC) orbiter warning: digest mismatch: after b0275bfe409329e90caa68828ca9cf99d935619c40adf20794b6b4a165f10dac != before 689768e222479cddedfe8605c20bdb7052e22d65868c907a97a7972f7764c1dd Duration: 841.796399ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [..X.............XX...XXXXX.XXXX.XXXXX..XXX.....XXX.X.X.XX.X.] duration: 49.207159857s FAIL (2025-07-26 22:56:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 3659698 (59h19m24.843958284s) [upload=56894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:54:53.895339399 (59h19m25.085637977s) [seg_pos={0 0}, upload=56894]: Begin upload segment... Jul 26 22:54:53.895397422 (59h19m25.085696005s) [seg_pos={0 0}, upload=56894]: Done begin upload segment. Jul 26 22:54:53.895422888 (59h19m25.08572147s) [noise=true , address=64.49.13.155:4918 , piece_id=X3J27AQVB66OB7TC3TAVIVXUXP2H36ADLV6EI3OOPKAUM64W5YGA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.895491522 (59h19m25.085790101s) [noise=true , address=45.91.116.9:4273 , piece_id=N5J6MYR4NEG5A4EXFEPEVNOI4EIJRY7QTCXBU5JMFZKEPVXJVTKA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.895957268 (59h19m25.08625585s) [noise=true , address=77.90.31.17:4784 , piece_id=25GWTVWEGSA7344OLNCYSPKVCVG3D6WWK3AUAT3TBLHX22XY37ZQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.896262941 (59h19m25.08656152s) [noise=true , address=64.49.13.26:4951 , piece_id=PQQBGTEP27MXZ3YY7L5GQJL6FX56GJVHBCI27ZB6OIHKMJRSIGLA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.896811576 (59h19m25.087110159s) [noise=true , address=64.49.13.140:4508 , piece_id=DGUUC5ABLL7TBJEGUBLMIJFMWJQ76E3Q4KHTOXF3CJJI7UKDPV5Q, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897286536 (59h19m25.087585115s) [noise=true , address=79.127.219.38:4006 , piece_id=T5L2FDRKRWYTQGBNF7UAR774AEA47GI4OXX7KBNHMCRNNTLNB4XA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897379252 (59h19m25.087677838s) [noise=true , address=45.91.116.14:4194 , piece_id=FOWBWZJQ2QN7KD3AK5UKTW2CWKARYW3HADAL46CO3IAXTSKIZSYQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897758399 (59h19m25.088056978s) [noise=true , address=64.49.13.140:4514 , piece_id=YRPYCOA7MALDIETIMT3ZCWNRDBQ34ZAG6J4E4X2REVUD7G43VXXQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.897766108 (59h19m25.088064693s) [noise=true , address=79.127.201.215:4352 , piece_id=4JXTKYGEZKITND4KEEXOJKGFCNBFH5M7O55A65RTIVYRDAMNJZHA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898175302 (59h19m25.088473884s) [noise=true , address=45.91.116.14:4206 , piece_id=FXPC4WAH2MYZXAE2JW47WFMCJWA7CSGYJZ3FTAETJHOI35Y7T3VA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898282568 (59h19m25.08858115s) [noise=true , address=77.90.31.15:4703 , piece_id=77BNC2ZFGXN3X67AW7JFH2555EJJB4UCWADXZUE6AL7UVZ2JGVUA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898532129 (59h19m25.088830715s) [noise=true , address=79.127.205.233:4629 , piece_id=37HF3OWKWH2FHKPRHWWJ5N7YNCIMWWPCI7QBZV4ZBOBC62X2EUHA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898745859 (59h19m25.089044441s) [noise=true , address=79.127.220.101:4048 , piece_id=4KAJHO6NSQ7YGKYK3OMJU5DHN2ZDUIXRLBWHMFIDFHU42XTV5IEQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.898843412 (59h19m25.089141996s) [noise=true , address=45.91.116.11:4121 , piece_id=II7ELY7KO5IYEQKPPU4GXLWUAHWEGZT37HD3MKSK6SH2336GQ64Q, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899145378 (59h19m25.089443962s) [noise=true , address=79.127.219.51:4212 , piece_id=7DR6RDCRSWLGSL4LRCJHFA5X5ZXBKYRRE4XUC7R6W3HBGDECXYRQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899291753 (59h19m25.089590332s) [noise=true , address=77.90.31.19:4867 , piece_id=WWBDA2NJGZ4IY7O2EFKVVIMUG6IDQ6F7AB2ILIHEU23BAVOSVK4A, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899501827 (59h19m25.089800411s) [noise=true , address=79.127.205.246:4281 , piece_id=MBPFA3HHSDTWC3B6GKWNTW4HSUQNJXU6HQNBIVOC7GONWQAEUHEA, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:53.899699595 (59h19m25.089998174s) [noise=true , address=156.146.43.225:4449 , piece_id=KPISBKWXFORIUITPSSXGILEYCPXLMKSXQT6XBSTS7FVSXXUU7OYQ, seg_pos={0 0}, upload=56894]: Uploading piece... Jul 26 22:54:54.417137519 (59h19m25.607436113s) [noise=true , address=79.127.220.101:4048 , piece_id=4KAJHO6NSQ7YGKYK3OMJU5DHN2ZDUIXRLBWHMFIDFHU42XTV5IEQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:54.454141381 (59h19m25.644439961s) [noise=true , address=156.146.43.225:4449 , piece_id=KPISBKWXFORIUITPSSXGILEYCPXLMKSXQT6XBSTS7FVSXXUU7OYQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:54.457574476 (59h19m25.647873113s) [upload=56894]: Next returned nil segment Jul 26 22:54:54.457593123 (59h19m25.647891701s) [upload=56894]: Waiting for error group managing segments... Jul 26 22:54:54.615981623 (59h19m25.806280202s) [noise=true , address=64.49.13.155:4918 , piece_id=X3J27AQVB66OB7TC3TAVIVXUXP2H36ADLV6EI3OOPKAUM64W5YGA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK, address:64.49.13.155:4918): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:54.620288789 (59h19m25.810587358s) [noise=true , address=45.91.116.14:4206 , piece_id=FXPC4WAH2MYZXAE2JW47WFMCJWA7CSGYJZ3FTAETJHOI35Y7T3VA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:55.155310877 (59h19m26.34560946s) [noise=true , address=64.49.13.26:4951 , piece_id=PQQBGTEP27MXZ3YY7L5GQJL6FX56GJVHBCI27ZB6OIHKMJRSIGLA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:55.194851814 (59h19m26.385150395s) [noise=true , address=79.127.205.246:4281 , piece_id=MBPFA3HHSDTWC3B6GKWNTW4HSUQNJXU6HQNBIVOC7GONWQAEUHEA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:55.214815830 (59h19m26.4051144s) [noise=true , address=79.127.205.233:4629 , piece_id=37HF3OWKWH2FHKPRHWWJ5N7YNCIMWWPCI7QBZV4ZBOBC62X2EUHA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:55.792814728 (59h19m26.983113317s) [noise=true , address=45.91.116.9:4273 , piece_id=N5J6MYR4NEG5A4EXFEPEVNOI4EIJRY7QTCXBU5JMFZKEPVXJVTKA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:56.233265946 (59h19m27.423564525s) [noise=true , address=79.127.219.38:4006 , piece_id=T5L2FDRKRWYTQGBNF7UAR774AEA47GI4OXX7KBNHMCRNNTLNB4XA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:56.269645596 (59h19m27.459944168s) [noise=true , address=77.90.31.15:4703 , piece_id=77BNC2ZFGXN3X67AW7JFH2555EJJB4UCWADXZUE6AL7UVZ2JGVUA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:57.940034816 (59h19m29.130333398s) [noise=true , address=79.127.219.51:4212 , piece_id=7DR6RDCRSWLGSL4LRCJHFA5X5ZXBKYRRE4XUC7R6W3HBGDECXYRQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:58.788209768 (59h19m29.978508336s) [noise=true , address=64.49.13.140:4514 , piece_id=YRPYCOA7MALDIETIMT3ZCWNRDBQ34ZAG6J4E4X2REVUD7G43VXXQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:54:59.398611894 (59h19m30.588910459s) [noise=true , address=45.91.116.11:4121 , piece_id=II7ELY7KO5IYEQKPPU4GXLWUAHWEGZT37HD3MKSK6SH2336GQ64Q, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:55:05.791624622 (59h19m36.981923195s) [noise=true , address=64.49.13.140:4508 , piece_id=DGUUC5ABLL7TBJEGUBLMIJFMWJQ76E3Q4KHTOXF3CJJI7UKDPV5Q, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:55:43.211019053 (59h20m14.401317622s) [noise=true , address=79.127.201.215:4352 , piece_id=4JXTKYGEZKITND4KEEXOJKGFCNBFH5M7O55A65RTIVYRDAMNJZHA, seg_pos={0 0}, upload=56894]: Done uploading piece. err: <nil> Jul 26 22:55:53.372353686 (59h20m24.562652275s) [noise=true , address=77.90.31.19:4867 , piece_id=WWBDA2NJGZ4IY7O2EFKVVIMUG6IDQ6F7AB2ILIHEU23BAVOSVK4A, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:55:53.372645226 (59h20m24.562943805s) [noise=true , address=77.90.31.17:4784 , piece_id=25GWTVWEGSA7344OLNCYSPKVCVG3D6WWK3AUAT3TBLHX22XY37ZQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: write tcp 139.59.144.22:33738->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:55:53.372722799 (59h20m24.563021379s) [noise=true , address=45.91.116.14:4194 , piece_id=FOWBWZJQ2QN7KD3AK5UKTW2CWKARYW3HADAL46CO3IAXTSKIZSYQ, seg_pos={0 0}, upload=56894]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: write tcp 139.59.144.22:50024->45.91.116.14:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:55:53.372757166 (59h20m24.563055752s) [seg_pos={0 0}, upload=56894]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:55:53.478852984 (59h20m24.669151559s) [upload=56894]: Done uploading segments... Jul 26 22:55:53.478900783 (59h20m24.669199364s) [upload=56894]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 49.207180903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...........X.......X..X...X.XX...X..X.XX......X.X.......X...] duration: 9.05344232s FAIL (2025-07-26 22:56:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.053481938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.300445864s FAIL (2025-07-26 22:56:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 988 (58h42m45.061082575s) [upload=56308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:18:14.018433272 (58h42m45.208731856s) [upload=56308]: Done uploading segments... Jul 26 22:18:14.018484438 (58h42m45.208783019s) [upload=56308]: Upload finished. err: <nil> Jul 26 22:18:57.520834389 (58h43m28.711132968s) [upload=56321]: Starting upload Jul 26 22:18:57.520934224 (58h43m28.711232794s) [upload=56321]: Done starting upload Jul 26 22:18:57.521309042 (58h43m28.711607621s) [upload=56321]: Uploading segments... Jul 26 22:18:57.521389036 (58h43m28.711687618s) [upload=56321]: Got next segment. Inline: false Jul 26 22:18:57.521430569 (58h43m28.711729148s) [upload=56321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:18:57.521435990 (58h43m28.711734568s) [upload=56321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:18:57.634354903 (58h43m28.824653493s) [seg_pos={0 0}, upload=56321]: Begin upload segment... Jul 26 22:18:57.634433002 (58h43m28.824731593s) [seg_pos={0 0}, upload=56321]: Done begin upload segment. Jul 26 22:18:57.634466006 (58h43m28.824764596s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.634507350 (58h43m28.82480593s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635011032 (58h43m28.825309612s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635154508 (58h43m28.825453097s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635425419 (58h43m28.825724018s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635823171 (58h43m28.826121762s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.635848858 (58h43m28.826147438s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636411603 (58h43m28.826710186s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636540327 (58h43m28.826838918s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.636837624 (58h43m28.827136209s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637107959 (58h43m28.827406542s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637209450 (58h43m28.827508033s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.637657116 (58h43m28.827955704s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638167018 (58h43m28.828465602s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.638722599 (58h43m28.829021185s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:57.639240448 (58h43m28.82953902s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Uploading piece... Jul 26 22:18:58.247247885 (58h43m29.437546466s) [upload=56321]: Next returned nil segment Jul 26 22:18:58.247273951 (58h43m29.437572537s) [upload=56321]: Waiting for error group managing segments... Jul 26 22:18:58.383663350 (58h43m29.573961931s) [noise=true , address=45.91.116.13:4154 , piece_id=PNBIV7A27TASCKR2CEQT7NJ5ZTG3NUQ2PLXJH7R47BFQQ53FLKUQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384857324 (58h43m29.575155903s) [noise=true , address=45.91.116.8:4047 , piece_id=YTLRWKOOJ4XIGA3SKHFT2K66R57LBTW7OOVZSVJMCCV3YISPGWLA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.384981909 (58h43m29.575280495s) [noise=true , address=45.91.116.6:4973 , piece_id=S5BZKUNC2KNOVQMFRHSXOZNZZG6BUBK57ECM6W3S4YKN55KPOSFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.387318149 (58h43m29.577616722s) [noise=true , address=45.91.116.7:4009 , piece_id=2JHIJGUSNOSVSGOSGHHIYA2RNVLRS2P7MIKYHLSX4PTQOWNZKKGQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.395512660 (58h43m29.585811243s) [noise=true , address=45.91.116.12:4227 , piece_id=FICCSE6IJQKUGYKTS3JHH4G65CPBOGDSPVIYXT25NR2GKRZQLOXA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.401895917 (58h43m29.592194499s) [noise=true , address=45.91.116.10:4081 , piece_id=2R44TDAMR2O3CNYFRZXOCSMG2ISUWMIQPVDWN3V2ECZWW4JUHNMA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.402135621 (58h43m29.59243419s) [noise=true , address=77.90.31.11:4553 , piece_id=BP6FPURAP7D6RT34G72KZ32UY76QKXDG464WKNDPNUNOD6WS3KFQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.403028120 (58h43m29.593326693s) [noise=true , address=45.91.116.8:4048 , piece_id=BF2LQK3BFGLPNCTDWB2M6HLRBSYBCXLR2AO7CMPERHT4DIBNKC4Q, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.404431307 (58h43m29.594729891s) [noise=true , address=45.91.116.4:4938 , piece_id=INUA3CJT7KOAKKRYSBLTNXLX3MKWYYAFZ45BMUOZ6Y6I3B2EHHCQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:58.409536042 (58h43m29.599834622s) [noise=true , address=45.91.116.7:4010 , piece_id=DZSLTEUQELXNHSH57IUZ6PRMOS56GR5TCWNAEEY7Y6YJ7KNDTRJA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:18:59.425708084 (58h43m30.616006681s) [noise=true , address=77.90.31.21:4877 , piece_id=RCTFN3VAXG26GTA6GFQDQDIBAHJPOSI3QUSLLKO5APYYA57BSBRQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:00.437139003 (58h43m31.627437582s) [noise=true , address=77.90.31.12:4590 , piece_id=PZSNEJOE623OVLNPZSRVVTTAGNHGRD7JCBUXCB3FXOMUE4CQNLFA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: <nil> Jul 26 22:19:27.246703337 (58h43m58.437001916s) [noise=true , address=104.36.82.7:4909 , piece_id=HOFM67WVGW5QXUA65MEPQLJF7EV25AXBG4GNMREFYF2RNRGH2XWQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57728->104.36.82.7:4909: use of closed network connection Jul 26 22:19:27.246791447 (58h43m58.437090027s) [noise=true , address=104.36.82.11:4114 , piece_id=ZUBONJLMV4QRTG7DLIHWZICNNQ5OOF3HAPN26Q5AMK4TY4OYTXNA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44152->104.36.82.11:4114: use of closed network connection Jul 26 22:19:27.246848250 (58h43m58.437146829s) [noise=true , address=104.36.82.3:4767 , piece_id=IWRMBBOMGH7ZV3JA326TJ4SWHLEGA2T6CDLUDKJ4RFZBUZMTA3MQ, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52976->104.36.82.3:4767: use of closed network connection Jul 26 22:19:27.246885606 (58h43m58.4371843s) [noise=true , address=104.36.82.12:4150 , piece_id=4PVRPBKTUNFKUWDJDTXYCIXDKONWG5OPQVB2M6KPKK66SMCYYHGA, seg_pos={0 0}, upload=56321]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43452->104.36.82.12:4150: use of closed network connection Jul 26 22:19:27.246926689 (58h43m58.437225269s) [seg_pos={0 0}, upload=56321]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:19:27.354450344 (58h43m58.544748918s) [upload=56321]: Done uploading segments... Jul 26 22:19:27.354479247 (58h43m58.544777827s) [upload=56321]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 2.300475335s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)