Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......................X.X..........................] duration: 998.748738ms FAIL (2025-07-27 00:33:24 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 998.782039ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX......XXXXX] duration: 30.110336643s FAIL (2025-07-27 00:33:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 72581757s) [upload=58411]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:32:52.211002055 (60h57m23.401300625s) [upload=58425]: Starting upload Jul 27 00:32:52.211082477 (60h57m23.401381061s) [upload=58425]: Done starting upload Jul 27 00:32:52.211226092 (60h57m23.401524672s) [upload=58425]: Uploading segments... Jul 27 00:32:52.211562829 (60h57m23.401861408s) [upload=58425]: Got next segment. Inline: false Jul 27 00:32:52.211607167 (60h57m23.401905736s) [upload=58425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:32:52.211613189 (60h57m23.401911768s) [upload=58425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:32:52.322597518 (60h57m23.5128961s) [seg_pos={0 0}, upload=58425]: Begin upload segment... Jul 27 00:32:52.322642694 (60h57m23.512941264s) [seg_pos={0 0}, upload=58425]: Done begin upload segment. Jul 27 00:32:52.322670948 (60h57m23.51296953s) [noise=true , address=45.91.116.14:4191 , piece_id=DI5DKVTAW4VKMCHWS5OML6Y27MHFOLM2YKTDLU47YI3PHHXE2WHQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.322676738 (60h57m23.512975317s) [noise=true , address=45.91.116.12:4225 , piece_id=76K4RIJFIRGELZRBHW6YPTLQFJN3SKX7VPTSDY6LIJ53LGRI254A, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.323071084 (60h57m23.513369667s) [noise=true , address=104.36.82.6:4876 , piece_id=FXQKMB7WCZ7ZYQAN3HVV7VKZG5N4C3EMEBEKJG4E6INQHVGUTLKA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.323394609 (60h57m23.513693189s) [noise=true , address=77.90.31.16:4735 , piece_id=DC4XLTY7JLILN5CEK7LLDT5Y6DGXMFPJVEVKCESHMMCUQPXILGSA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.323633707 (60h57m23.513932287s) [noise=true , address=45.91.116.7:4009 , piece_id=NSTOCISKCVWIB5RU54XKVIDHGE3CLVETUBUY6JVS3CDIH2VDCRKQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.323695085 (60h57m23.513993658s) [noise=true , address=104.36.82.5:4837 , piece_id=NMCCMGWPII5QMMVKDBMB4SKPSVD4NSKUBPP2QMZEA6RJL3QZRXJA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.323970590 (60h57m23.514269169s) [noise=true , address=104.36.82.16:4295 , piece_id=OEGPA3SIQXZMIN7N5KQXWFU3ELHDISCPPID2HE2DSRGSNYT5PYJA, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.324260573 (60h57m23.514559155s) [noise=true , address=45.91.116.4:4939 , piece_id=Y5UYNHG24XP5TTXBRJB6YSPSBGIVOZWCGKZCTGTQZHSJBHCX5NCQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.324534076 (60h57m23.514832659s) [noise=true , address=77.90.31.14:4661 , piece_id=3IOAZNUTSQCWEQ2P7LWLGKQBREXRMUBKJPEKUXHGJK72T4MATDKQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.324782978 (60h57m23.515081566s) [noise=true , address=45.91.116.9:4263 , piece_id=ELFKELMTPYRXVMZO4VZLNGV2JZ3LG5TUVGGLRDETDC7AGORL63PQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.325279317 (60h57m23.515577904s) [noise=true , address=45.91.116.8:4045 , piece_id=6YC3HIHQUI73ZFWMK35QAI2QLUCTWKG2CT53XN42L64YRBZBYBOQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.325636727 (60h57m23.515935312s) [noise=true , address=104.36.82.12:4151 , piece_id=NRUJVT6XPUICYBFKELR6FLFGXIXSVP5DAKPP4QXCCOJB3TVUGJCQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.326032022 (60h57m23.516330608s) [noise=true , address=104.36.82.4:4802 , piece_id=33EN6YFFELNTHYIZRABNRXG6Q2FC7LLKAXMJTJPTNPU27RDTCINQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.326159701 (60h57m23.516458289s) [noise=true , address=45.91.116.5:4299 , piece_id=VX2HKAI5CONUA57P3B6YMKVX4M4XPTEA4ZQQYLGSMRHOZFSUWLYQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.326520245 (60h57m23.516818832s) [noise=true , address=45.91.116.13:4153 , piece_id=QVLJWP7A6ZSJVLULKIR3QUIL6LX6YPE75Y4LJ7NXKXQRW3PRCTNQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.326567249 (60h57m23.516865831s) [noise=true , address=104.36.82.14:4221 , piece_id=5DYTGZTPP6H4YFO3PJF6JNNHOR2MBYGDIXLV2YU4OG4J34TK5VHQ, seg_pos={0 0}, upload=58425]: Uploading piece... Jul 27 00:32:52.933620010 (60h57m24.123918582s) [upload=58425]: Next returned nil segment Jul 27 00:32:52.933643836 (60h57m24.123942425s) [upload=58425]: Waiting for error group managing segments... Jul 27 00:32:53.069586610 (60h57m24.259885183s) [noise=true , address=45.91.116.8:4045 , piece_id=6YC3HIHQUI73ZFWMK35QAI2QLUCTWKG2CT53XN42L64YRBZBYBOQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.073544590 (60h57m24.263843174s) [noise=true , address=45.91.116.4:4939 , piece_id=Y5UYNHG24XP5TTXBRJB6YSPSBGIVOZWCGKZCTGTQZHSJBHCX5NCQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.074681827 (60h57m24.264980425s) [noise=true , address=45.91.116.5:4299 , piece_id=VX2HKAI5CONUA57P3B6YMKVX4M4XPTEA4ZQQYLGSMRHOZFSUWLYQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.076446806 (60h57m24.266745378s) [noise=true , address=45.91.116.9:4263 , piece_id=ELFKELMTPYRXVMZO4VZLNGV2JZ3LG5TUVGGLRDETDC7AGORL63PQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.083809253 (60h57m24.274107823s) [noise=true , address=77.90.31.16:4735 , piece_id=DC4XLTY7JLILN5CEK7LLDT5Y6DGXMFPJVEVKCESHMMCUQPXILGSA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.087101041 (60h57m24.277399614s) [noise=true , address=45.91.116.13:4153 , piece_id=QVLJWP7A6ZSJVLULKIR3QUIL6LX6YPE75Y4LJ7NXKXQRW3PRCTNQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.108197432 (60h57m24.298496008s) [noise=true , address=77.90.31.14:4661 , piece_id=3IOAZNUTSQCWEQ2P7LWLGKQBREXRMUBKJPEKUXHGJK72T4MATDKQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.111213970 (60h57m24.301512543s) [noise=true , address=45.91.116.7:4009 , piece_id=NSTOCISKCVWIB5RU54XKVIDHGE3CLVETUBUY6JVS3CDIH2VDCRKQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.163133179 (60h57m24.353431752s) [noise=true , address=45.91.116.14:4191 , piece_id=DI5DKVTAW4VKMCHWS5OML6Y27MHFOLM2YKTDLU47YI3PHHXE2WHQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:32:53.166970078 (60h57m24.35726865s) [noise=true , address=45.91.116.12:4225 , piece_id=76K4RIJFIRGELZRBHW6YPTLQFJN3SKX7VPTSDY6LIJ53LGRI254A, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:33:08.142639727 (60h57m39.332938307s) [noise=true , address=104.36.82.16:4295 , piece_id=OEGPA3SIQXZMIN7N5KQXWFU3ELHDISCPPID2HE2DSRGSNYT5PYJA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: <nil> Jul 27 00:33:21.924890949 (60h57m53.115189585s) [noise=true , address=104.36.82.4:4802 , piece_id=33EN6YFFELNTHYIZRABNRXG6Q2FC7LLKAXMJTJPTNPU27RDTCINQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:33:21.925257478 (60h57m53.115556047s) [noise=true , address=104.36.82.6:4876 , piece_id=FXQKMB7WCZ7ZYQAN3HVV7VKZG5N4C3EMEBEKJG4E6INQHVGUTLKA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:38306->104.36.82.6:4876: use of closed network connection; write tcp 139.59.144.22:38306->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38306->104.36.82.6:4876: use of closed network connection Jul 27 00:33:21.925326615 (60h57m53.115625184s) [noise=true , address=104.36.82.5:4837 , piece_id=NMCCMGWPII5QMMVKDBMB4SKPSVD4NSKUBPP2QMZEA6RJL3QZRXJA, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:57280->104.36.82.5:4837: use of closed network connection; write tcp 139.59.144.22:57280->104.36.82.5:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57280->104.36.82.5:4837: use of closed network connection Jul 27 00:33:21.925380055 (60h57m53.115678624s) [noise=true , address=104.36.82.14:4221 , piece_id=5DYTGZTPP6H4YFO3PJF6JNNHOR2MBYGDIXLV2YU4OG4J34TK5VHQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 139.59.144.22:57046->104.36.82.14:4221: use of closed network connection; write tcp 139.59.144.22:57046->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57046->104.36.82.14:4221: use of closed network connection Jul 27 00:33:21.925427532 (60h57m53.115726101s) [noise=true , address=104.36.82.12:4151 , piece_id=NRUJVT6XPUICYBFKELR6FLFGXIXSVP5DAKPP4QXCCOJB3TVUGJCQ, seg_pos={0 0}, upload=58425]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 139.59.144.22:38068->104.36.82.12:4151: use of closed network connection; write tcp 139.59.144.22:38068->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38068->104.36.82.12:4151: use of closed network connection Jul 27 00:33:21.925445685 (60h57m53.115744264s) [seg_pos={0 0}, upload=58425]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:33:22.031124019 (60h57m53.221422591s) [upload=58425]: Done uploading segments... Jul 27 00:33:22.031173847 (60h57m53.22147243s) [upload=58425]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.110366059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........X.......................X.X.........................] duration: 884.194583ms FAIL (2025-07-27 00:33:03 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 884.237833ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X..X............X.X..X.X...............................] duration: 40.679584082s FAIL (2025-07-27 00:32:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.679607665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.....X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X.] duration: 33.79959073s FAIL (2025-07-27 00:32:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== (60h55m44.610356365s) [upload=58403]: Done starting upload Jul 27 00:31:13.420316232 (60h55m44.610614821s) [upload=58403]: Uploading segments... Jul 27 00:31:13.420397592 (60h55m44.610696177s) [upload=58403]: Got next segment. Inline: false Jul 27 00:31:13.420454603 (60h55m44.610753199s) [upload=58403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:13.420461977 (60h55m44.610760567s) [upload=58403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:13.613432370 (60h55m44.803730949s) [seg_pos={0 0}, upload=58403]: Begin upload segment... Jul 27 00:31:13.613485653 (60h55m44.803784218s) [seg_pos={0 0}, upload=58403]: Done begin upload segment. Jul 27 00:31:13.613513779 (60h55m44.803812365s) [noise=true , address=104.36.82.47:4549 , piece_id=QYEFJCAO2JL2V7GOXSYIKDI3KHXZOSOCAYDX36AAJMOFTTD5H2JQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.613531740 (60h55m44.803830323s) [noise=true , address=79.127.205.246:4282 , piece_id=TKPEGPDUQFDTFAGBBACDVSNGG5E7IG7ZBAYI3JNSF72ZFUNYCQRQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.613850734 (60h55m44.804149313s) [noise=true , address=79.127.205.245:4226 , piece_id=5QPQWQ34XGXPUT7MVXJZF7DY6RW6PM2247PGAKHWGNT2BTYJ5SSA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614121664 (60h55m44.804420246s) [noise=true , address=45.91.116.14:4206 , piece_id=HOPWK6GRYV7JG3OUX2QMAJ2B75OVR4J36QO7KCFU5RQKE54UQSJQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614202980 (60h55m44.80450156s) [noise=true , address=77.90.31.18:4821 , piece_id=VA32SCYMUL3WCDV7ONS4FTF6ZWRZKMSRUALP22V273QEUWOYD56A, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614352038 (60h55m44.804650617s) [noise=true , address=104.36.82.56:4238 , piece_id=WZPLGWBE4KING7ENAG7YHEWXR5EY56CUL2YXRLK275446YKXLSSA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614509461 (60h55m44.804808043s) [noise=true , address=104.36.82.21:4499 , piece_id=S2R2DQWWHTLLTWPKGAHCNWS6BLQXQLO2VWYBC2RF27WA7NABFBBA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614614918 (60h55m44.8049135s) [noise=true , address=79.127.205.234:4619 , piece_id=EZWLZOEWF6FC625TPK4PE6PM2PJB37ILDVRZ6M4L4GWG5T54ESCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614790069 (60h55m44.805088652s) [noise=true , address=79.127.205.235:4673 , piece_id=RDMWTVWFFJUIVMUVECOLHD44KSBUTOPYH4QGP5XYYYQ4A3KCBMFA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614902348 (60h55m44.805200933s) [noise=true , address=104.36.82.46:4490 , piece_id=MV72INCBFO3CDX5MMKYDKRVES2KGRPZMHLP77DJSALAAIKXUXO3Q, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615182669 (60h55m44.805481251s) [noise=true , address=77.90.31.19:4871 , piece_id=46PZAK3YFMAK2XCAVBAYREWN4Q7Y6BYR7P5WHZBE5E5VKLIWY2UQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615304517 (60h55m44.805603118s) [noise=true , address=77.90.31.10:4538 , piece_id=6YRKWWJMF47ODBRLHSDJXI2IOR6VYRNKMYLC4ALHMZZ2EMJF62BQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615625009 (60h55m44.805923588s) [noise=true , address=79.127.201.215:4356 , piece_id=JRZJEAU3VMXU57T4G6ADAMKCH7M2TYSZUTEQZFTAITG2OZESTWCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615655019 (60h55m44.805953665s) [noise=true , address=79.127.205.233:4643 , piece_id=SJIKZ5G2MMHLN2UESQTC2RV7XZRXEBSLZAIOHVKVJAKCY653PNEA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615915115 (60h55m44.806213684s) [noise=true , address=79.127.219.38:4006 , piece_id=3LXW7MHZ3FRTV3DJW6KKSHFJSY2Y6HSW2DFBQ3WQYSKVWN4SVCVA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.616230312 (60h55m44.806528892s) [noise=true , address=104.36.82.53:4132 , piece_id=2QDVBG4PLYCWEBOJJVIWULIXBVFXPYQGKSMTYVWZ4ZXAQ5VZCJYQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.616506333 (60h55m44.806804912s) [noise=true , address=77.90.31.15:4722 , piece_id=O7YQIFEWMUWHW2HZENCBREBWMHNB5E7C6UQDIPRGQ2HKGWSZXRCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.616783194 (60h55m44.807081774s) [noise=true , address=79.127.201.214:4289 , piece_id=ZQVBPCLNOYPO4JFH4QHF3SBGYT7OEWJDDOYNHR7PTVASXKKSMPCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:14.142368574 (60h55m45.332667141s) [noise=true , address=79.127.201.214:4289 , piece_id=ZQVBPCLNOYPO4JFH4QHF3SBGYT7OEWJDDOYNHR7PTVASXKKSMPCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:31:14.178859025 (60h55m45.369157596s) [upload=58403]: Next returned nil segment Jul 27 00:31:14.178883220 (60h55m45.369181799s) [upload=58403]: Waiting for error group managing segments... Jul 27 00:31:14.793294993 (60h55m45.983593582s) [noise=true , address=79.127.219.38:4006 , piece_id=3LXW7MHZ3FRTV3DJW6KKSHFJSY2Y6HSW2DFBQ3WQYSKVWN4SVCVA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:14.906742298 (60h55m46.097040867s) [noise=true , address=79.127.205.245:4226 , piece_id=5QPQWQ34XGXPUT7MVXJZF7DY6RW6PM2247PGAKHWGNT2BTYJ5SSA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:14.998131187 (60h55m46.188429758s) [noise=true , address=79.127.205.246:4282 , piece_id=TKPEGPDUQFDTFAGBBACDVSNGG5E7IG7ZBAYI3JNSF72ZFUNYCQRQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.030463698 (60h55m46.22076228s) [noise=true , address=79.127.205.235:4673 , piece_id=RDMWTVWFFJUIVMUVECOLHD44KSBUTOPYH4QGP5XYYYQ4A3KCBMFA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.110956602 (60h55m46.301255174s) [noise=true , address=79.127.205.233:4643 , piece_id=SJIKZ5G2MMHLN2UESQTC2RV7XZRXEBSLZAIOHVKVJAKCY653PNEA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.267600997 (60h55m46.457899563s) [noise=true , address=77.90.31.10:4538 , piece_id=6YRKWWJMF47ODBRLHSDJXI2IOR6VYRNKMYLC4ALHMZZ2EMJF62BQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.546649037 (60h55m46.736947611s) [noise=true , address=79.127.205.234:4619 , piece_id=EZWLZOEWF6FC625TPK4PE6PM2PJB37ILDVRZ6M4L4GWG5T54ESCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:16.052389924 (60h55m47.242688504s) [noise=true , address=45.91.116.14:4206 , piece_id=HOPWK6GRYV7JG3OUX2QMAJ2B75OVR4J36QO7KCFU5RQKE54UQSJQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:19.609570723 (60h55m50.799869297s) [noise=true , address=77.90.31.19:4871 , piece_id=46PZAK3YFMAK2XCAVBAYREWN4Q7Y6BYR7P5WHZBE5E5VKLIWY2UQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:26.137992383 (60h55m57.328290964s) [noise=true , address=79.127.201.215:4356 , piece_id=JRZJEAU3VMXU57T4G6ADAMKCH7M2TYSZUTEQZFTAITG2OZESTWCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:34.085410066 (60h56m5.27570864s) [noise=true , address=104.36.82.56:4238 , piece_id=WZPLGWBE4KING7ENAG7YHEWXR5EY56CUL2YXRLK275446YKXLSSA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:52.466630613 (60h56m23.656929184s) [noise=true , address=104.36.82.47:4549 , piece_id=QYEFJCAO2JL2V7GOXSYIKDI3KHXZOSOCAYDX36AAJMOFTTD5H2JQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:32:03.443345661 (60h56m34.633644233s) [noise=true , address=104.36.82.46:4490 , piece_id=MV72INCBFO3CDX5MMKYDKRVES2KGRPZMHLP77DJSALAAIKXUXO3Q, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:32:13.149943633 (60h56m44.34024222s) [noise=true , address=77.90.31.18:4821 , piece_id=VA32SCYMUL3WCDV7ONS4FTF6ZWRZKMSRUALP22V273QEUWOYD56A, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:32:13.150052881 (60h56m44.340351461s) [noise=true , address=104.36.82.21:4499 , piece_id=S2R2DQWWHTLLTWPKGAHCNWS6BLQXQLO2VWYBC2RF27WA7NABFBBA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:1218y8DtJNTTu94dbqGhywmHR4s99zcz9issmMmpgwusr2KgZxc, address:104.36.82.21:4499): protocol: write tcp 139.59.144.22:35346->104.36.82.21:4499: use of closed network connection; write tcp 139.59.144.22:35346->104.36.82.21:4499: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35346->104.36.82.21:4499: use of closed network connection Jul 27 00:32:13.150145509 (60h56m44.340444089s) [noise=true , address=104.36.82.53:4132 , piece_id=2QDVBG4PLYCWEBOJJVIWULIXBVFXPYQGKSMTYVWZ4ZXAQ5VZCJYQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:32:13.150225277 (60h56m44.340523857s) [noise=true , address=77.90.31.15:4722 , piece_id=O7YQIFEWMUWHW2HZENCBREBWMHNB5E7C6UQDIPRGQ2HKGWSZXRCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:13.150256167 (60h56m44.34055475s) [seg_pos={0 0}, upload=58403]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 27 00:32:13.256444579 (60h56m44.446743162s) [upload=58403]: Done uploading segments... Jul 27 00:32:13.256475069 (60h56m44.44677365s) [upload=58403]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 33.799612996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X...X....X.......XX...X....X...XX....X......XXXX..XX...X....] duration: 4.596811996s FAIL (2025-07-27 00:32:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.596839491s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX......XXXX] duration: 30.110610468s FAIL (2025-07-27 00:32:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ssful: 10 optimal: 13 errs: <nil> Jul 27 00:31:44.251207851 (60h56m15.441506434s) [upload=58405]: Done uploading segments... Jul 27 00:31:44.251242595 (60h56m15.441541184s) [upload=58405]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:31:44.651861738 (60h56m15.842160326s) [upload=58411]: Starting upload Jul 27 00:31:44.651964664 (60h56m15.84226323s) [upload=58411]: Done starting upload Jul 27 00:31:44.652478897 (60h56m15.842777462s) [upload=58411]: Uploading segments... Jul 27 00:31:44.652544133 (60h56m15.842842722s) [upload=58411]: Got next segment. Inline: false Jul 27 00:31:44.652586074 (60h56m15.842884657s) [upload=58411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:44.652590978 (60h56m15.842889543s) [upload=58411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:44.765824924 (60h56m15.956123504s) [seg_pos={0 0}, upload=58411]: Begin upload segment... Jul 27 00:31:44.765876741 (60h56m15.956175307s) [seg_pos={0 0}, upload=58411]: Done begin upload segment. Jul 27 00:31:44.765939612 (60h56m15.956238191s) [noise=true , address=104.36.82.5:4838 , piece_id=4VDJD7CKKKJDJOYAKNQIJ4GOIUNYMWTEI34UF6JL6ZYVXNCFPTUQ, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.765978933 (60h56m15.956277514s) [noise=true , address=104.36.82.10:4079 , piece_id=4D2GQQF3BU2S6VSLY6FMKNADCO527LDDIL75YE67HXC32XXCDIIA, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.766304857 (60h56m15.956603439s) [noise=true , address=45.91.116.11:4118 , piece_id=DNDA4I4ZU4SCJT4QKH33764TN5NNGTKVIQARLBGUCFGFS5VWBFJA, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.766349189 (60h56m15.956647772s) [noise=true , address=104.36.82.16:4293 , piece_id=BLDM4OMAAV7NG2MMAN347HPN32S7FWQBCNZK6ACDERYKC2SNMR3Q, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.766589849 (60h56m15.956888431s) [noise=true , address=104.36.82.14:4222 , piece_id=5AGERPNMSFNXIZEIAS5M2PN3JMG7ZSPIYVYY7LIPWDQ7TRJAKY3Q, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.766723270 (60h56m15.957021859s) [noise=true , address=77.90.31.13:4627 , piece_id=3RJWV7CGE36WP3K5KSDOBM6UWUJFBNJOSDKZHUQN2BXYIINAYU2A, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.766881320 (60h56m15.957179899s) [noise=true , address=77.90.31.19:4841 , piece_id=7IVFDXWFOYMFOF4SKNOQI6LFPZ3CQ6IR6Q7ARKQJBF72BX2MO5HQ, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.767161810 (60h56m15.957460397s) [noise=true , address=104.36.82.4:4804 , piece_id=WVQMSH2UJQVPPTBDXFJ55CZMPCGTS52OMKVU7AAGZ77VPNAKHB7Q, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.767178490 (60h56m15.957477069s) [noise=true , address=45.91.116.8:4048 , piece_id=WOTAWHWEMJBUEF3MPYWNBNEA2STBSMLJNN2O5XHEFTIA2T4YEK7Q, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.767466677 (60h56m15.957765259s) [noise=true , address=45.91.116.12:4225 , piece_id=G7RCCDHIZC5IXXBD6D45VOSGFDVXKNKLHKYN6LT6FEC3B77FPL5Q, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.767560900 (60h56m15.957859491s) [noise=true , address=45.91.116.13:4153 , piece_id=CYMZDPXJGGF3RS5UT5Q5AD2KNAHRJBSDNMF55QPWPSLBLX57TGYQ, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.767798633 (60h56m15.9580972s) [noise=true , address=45.91.116.4:4939 , piece_id=PUOOQ4N2B4IIVE4YPZCTLGOI6RVTFY5HPNRC7I5WE7NIV4YADDIQ, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.767928634 (60h56m15.958227222s) [noise=true , address=45.91.116.9:4261 , piece_id=H37VLCY5HBYBGVKCRFNVZQW6LXETQIZIYO6YD5UQVFTDRDCDRRPA, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.768221555 (60h56m15.958520136s) [noise=true , address=77.90.31.19:4842 , piece_id=TO4LDPKW5D2FF5C7OF2JI2YX7I6H75ZOPKNBEB3VM4J6G7VHGUWA, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.768497105 (60h56m15.958795687s) [noise=true , address=45.91.116.14:4190 , piece_id=2TGXRADQ6BNXTKTLDM56LVR6OBITFLH5SRORZKYBVU2WRIEFSFNQ, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:44.769543904 (60h56m15.959842484s) [noise=true , address=45.91.116.8:4045 , piece_id=OSZFUBRYV7KDXVHWNOM4IOPRLAYZ3HL4FGG5EQFNW4XWCVAB5BHA, seg_pos={0 0}, upload=58411]: Uploading piece... Jul 27 00:31:45.379738897 (60h56m16.570037468s) [upload=58411]: Next returned nil segment Jul 27 00:31:45.379755521 (60h56m16.5700541s) [upload=58411]: Waiting for error group managing segments... Jul 27 00:31:45.514219197 (60h56m16.704517783s) [noise=true , address=45.91.116.12:4225 , piece_id=G7RCCDHIZC5IXXBD6D45VOSGFDVXKNKLHKYN6LT6FEC3B77FPL5Q, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:45.522386805 (60h56m16.712685384s) [noise=true , address=45.91.116.8:4045 , piece_id=OSZFUBRYV7KDXVHWNOM4IOPRLAYZ3HL4FGG5EQFNW4XWCVAB5BHA, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:45.530925852 (60h56m16.721224422s) [noise=true , address=45.91.116.13:4153 , piece_id=CYMZDPXJGGF3RS5UT5Q5AD2KNAHRJBSDNMF55QPWPSLBLX57TGYQ, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:45.530993337 (60h56m16.721291905s) [noise=true , address=45.91.116.4:4939 , piece_id=PUOOQ4N2B4IIVE4YPZCTLGOI6RVTFY5HPNRC7I5WE7NIV4YADDIQ, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:45.543733066 (60h56m16.734031632s) [noise=true , address=45.91.116.11:4118 , piece_id=DNDA4I4ZU4SCJT4QKH33764TN5NNGTKVIQARLBGUCFGFS5VWBFJA, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:45.594422308 (60h56m16.784720899s) [noise=true , address=45.91.116.14:4190 , piece_id=2TGXRADQ6BNXTKTLDM56LVR6OBITFLH5SRORZKYBVU2WRIEFSFNQ, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:45.607350932 (60h56m16.797649521s) [noise=true , address=45.91.116.8:4048 , piece_id=WOTAWHWEMJBUEF3MPYWNBNEA2STBSMLJNN2O5XHEFTIA2T4YEK7Q, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:45.612669183 (60h56m16.80296775s) [noise=true , address=45.91.116.9:4261 , piece_id=H37VLCY5HBYBGVKCRFNVZQW6LXETQIZIYO6YD5UQVFTDRDCDRRPA, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:46.886993190 (60h56m18.077291758s) [noise=true , address=77.90.31.19:4842 , piece_id=TO4LDPKW5D2FF5C7OF2JI2YX7I6H75ZOPKNBEB3VM4J6G7VHGUWA, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:46.887074352 (60h56m18.077372919s) [noise=true , address=77.90.31.13:4627 , piece_id=3RJWV7CGE36WP3K5KSDOBM6UWUJFBNJOSDKZHUQN2BXYIINAYU2A, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:47.478975071 (60h56m18.669273642s) [noise=true , address=77.90.31.19:4841 , piece_id=7IVFDXWFOYMFOF4SKNOQI6LFPZ3CQ6IR6Q7ARKQJBF72BX2MO5HQ, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:31:58.921646925 (60h56m30.111945497s) [noise=true , address=104.36.82.5:4838 , piece_id=4VDJD7CKKKJDJOYAKNQIJ4GOIUNYMWTEI34UF6JL6ZYVXNCFPTUQ, seg_pos={0 0}, upload=58411]: Done uploading piece. err: <nil> Jul 27 00:32:14.373544096 (60h56m45.563842667s) [noise=true , address=104.36.82.4:4804 , piece_id=WVQMSH2UJQVPPTBDXFJ55CZMPCGTS52OMKVU7AAGZ77VPNAKHB7Q, seg_pos={0 0}, upload=58411]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 139.59.144.22:50638->104.36.82.4:4804: use of closed network connection; write tcp 139.59.144.22:50638->104.36.82.4:4804: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50638->104.36.82.4:4804: use of closed network connection Jul 27 00:32:14.373647066 (60h56m45.563945635s) [noise=true , address=104.36.82.10:4079 , piece_id=4D2GQQF3BU2S6VSLY6FMKNADCO527LDDIL75YE67HXC32XXCDIIA, seg_pos={0 0}, upload=58411]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:51148->104.36.82.10:4079: use of closed network connection; write tcp 139.59.144.22:51148->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51148->104.36.82.10:4079: use of closed network connection Jul 27 00:32:14.373704072 (60h56m45.564002651s) [noise=true , address=104.36.82.14:4222 , piece_id=5AGERPNMSFNXIZEIAS5M2PN3JMG7ZSPIYVYY7LIPWDQ7TRJAKY3Q, seg_pos={0 0}, upload=58411]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:50448->104.36.82.14:4222: use of closed network connection; write tcp 139.59.144.22:50448->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50448->104.36.82.14:4222: use of closed network connection Jul 27 00:32:14.373757615 (60h56m45.564056184s) [noise=true , address=104.36.82.16:4293 , piece_id=BLDM4OMAAV7NG2MMAN347HPN32S7FWQBCNZK6ACDERYKC2SNMR3Q, seg_pos={0 0}, upload=58411]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 139.59.144.22:48684->104.36.82.16:4293: use of closed network connection; write tcp 139.59.144.22:48684->104.36.82.16:4293: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48684->104.36.82.16:4293: use of closed network connection Jul 27 00:32:14.373773479 (60h56m45.564072061s) [seg_pos={0 0}, upload=58411]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:32:14.482239879 (60h56m45.67253845s) [upload=58411]: Done uploading segments... Jul 27 00:32:14.482283179 (60h56m45.672581757s) [upload=58411]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.110632298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [......X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..X] duration: 1m0.108698417s FAIL (2025-07-27 00:32:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== (60h55m44.610356365s) [upload=58403]: Done starting upload Jul 27 00:31:13.420316232 (60h55m44.610614821s) [upload=58403]: Uploading segments... Jul 27 00:31:13.420397592 (60h55m44.610696177s) [upload=58403]: Got next segment. Inline: false Jul 27 00:31:13.420454603 (60h55m44.610753199s) [upload=58403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:13.420461977 (60h55m44.610760567s) [upload=58403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:13.613432370 (60h55m44.803730949s) [seg_pos={0 0}, upload=58403]: Begin upload segment... Jul 27 00:31:13.613485653 (60h55m44.803784218s) [seg_pos={0 0}, upload=58403]: Done begin upload segment. Jul 27 00:31:13.613513779 (60h55m44.803812365s) [noise=true , address=104.36.82.47:4549 , piece_id=QYEFJCAO2JL2V7GOXSYIKDI3KHXZOSOCAYDX36AAJMOFTTD5H2JQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.613531740 (60h55m44.803830323s) [noise=true , address=79.127.205.246:4282 , piece_id=TKPEGPDUQFDTFAGBBACDVSNGG5E7IG7ZBAYI3JNSF72ZFUNYCQRQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.613850734 (60h55m44.804149313s) [noise=true , address=79.127.205.245:4226 , piece_id=5QPQWQ34XGXPUT7MVXJZF7DY6RW6PM2247PGAKHWGNT2BTYJ5SSA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614121664 (60h55m44.804420246s) [noise=true , address=45.91.116.14:4206 , piece_id=HOPWK6GRYV7JG3OUX2QMAJ2B75OVR4J36QO7KCFU5RQKE54UQSJQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614202980 (60h55m44.80450156s) [noise=true , address=77.90.31.18:4821 , piece_id=VA32SCYMUL3WCDV7ONS4FTF6ZWRZKMSRUALP22V273QEUWOYD56A, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614352038 (60h55m44.804650617s) [noise=true , address=104.36.82.56:4238 , piece_id=WZPLGWBE4KING7ENAG7YHEWXR5EY56CUL2YXRLK275446YKXLSSA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614509461 (60h55m44.804808043s) [noise=true , address=104.36.82.21:4499 , piece_id=S2R2DQWWHTLLTWPKGAHCNWS6BLQXQLO2VWYBC2RF27WA7NABFBBA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614614918 (60h55m44.8049135s) [noise=true , address=79.127.205.234:4619 , piece_id=EZWLZOEWF6FC625TPK4PE6PM2PJB37ILDVRZ6M4L4GWG5T54ESCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614790069 (60h55m44.805088652s) [noise=true , address=79.127.205.235:4673 , piece_id=RDMWTVWFFJUIVMUVECOLHD44KSBUTOPYH4QGP5XYYYQ4A3KCBMFA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.614902348 (60h55m44.805200933s) [noise=true , address=104.36.82.46:4490 , piece_id=MV72INCBFO3CDX5MMKYDKRVES2KGRPZMHLP77DJSALAAIKXUXO3Q, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615182669 (60h55m44.805481251s) [noise=true , address=77.90.31.19:4871 , piece_id=46PZAK3YFMAK2XCAVBAYREWN4Q7Y6BYR7P5WHZBE5E5VKLIWY2UQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615304517 (60h55m44.805603118s) [noise=true , address=77.90.31.10:4538 , piece_id=6YRKWWJMF47ODBRLHSDJXI2IOR6VYRNKMYLC4ALHMZZ2EMJF62BQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615625009 (60h55m44.805923588s) [noise=true , address=79.127.201.215:4356 , piece_id=JRZJEAU3VMXU57T4G6ADAMKCH7M2TYSZUTEQZFTAITG2OZESTWCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615655019 (60h55m44.805953665s) [noise=true , address=79.127.205.233:4643 , piece_id=SJIKZ5G2MMHLN2UESQTC2RV7XZRXEBSLZAIOHVKVJAKCY653PNEA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.615915115 (60h55m44.806213684s) [noise=true , address=79.127.219.38:4006 , piece_id=3LXW7MHZ3FRTV3DJW6KKSHFJSY2Y6HSW2DFBQ3WQYSKVWN4SVCVA, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.616230312 (60h55m44.806528892s) [noise=true , address=104.36.82.53:4132 , piece_id=2QDVBG4PLYCWEBOJJVIWULIXBVFXPYQGKSMTYVWZ4ZXAQ5VZCJYQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.616506333 (60h55m44.806804912s) [noise=true , address=77.90.31.15:4722 , piece_id=O7YQIFEWMUWHW2HZENCBREBWMHNB5E7C6UQDIPRGQ2HKGWSZXRCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:13.616783194 (60h55m44.807081774s) [noise=true , address=79.127.201.214:4289 , piece_id=ZQVBPCLNOYPO4JFH4QHF3SBGYT7OEWJDDOYNHR7PTVASXKKSMPCQ, seg_pos={0 0}, upload=58403]: Uploading piece... Jul 27 00:31:14.142368574 (60h55m45.332667141s) [noise=true , address=79.127.201.214:4289 , piece_id=ZQVBPCLNOYPO4JFH4QHF3SBGYT7OEWJDDOYNHR7PTVASXKKSMPCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:31:14.178859025 (60h55m45.369157596s) [upload=58403]: Next returned nil segment Jul 27 00:31:14.178883220 (60h55m45.369181799s) [upload=58403]: Waiting for error group managing segments... Jul 27 00:31:14.793294993 (60h55m45.983593582s) [noise=true , address=79.127.219.38:4006 , piece_id=3LXW7MHZ3FRTV3DJW6KKSHFJSY2Y6HSW2DFBQ3WQYSKVWN4SVCVA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:14.906742298 (60h55m46.097040867s) [noise=true , address=79.127.205.245:4226 , piece_id=5QPQWQ34XGXPUT7MVXJZF7DY6RW6PM2247PGAKHWGNT2BTYJ5SSA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:14.998131187 (60h55m46.188429758s) [noise=true , address=79.127.205.246:4282 , piece_id=TKPEGPDUQFDTFAGBBACDVSNGG5E7IG7ZBAYI3JNSF72ZFUNYCQRQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.030463698 (60h55m46.22076228s) [noise=true , address=79.127.205.235:4673 , piece_id=RDMWTVWFFJUIVMUVECOLHD44KSBUTOPYH4QGP5XYYYQ4A3KCBMFA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.110956602 (60h55m46.301255174s) [noise=true , address=79.127.205.233:4643 , piece_id=SJIKZ5G2MMHLN2UESQTC2RV7XZRXEBSLZAIOHVKVJAKCY653PNEA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.267600997 (60h55m46.457899563s) [noise=true , address=77.90.31.10:4538 , piece_id=6YRKWWJMF47ODBRLHSDJXI2IOR6VYRNKMYLC4ALHMZZ2EMJF62BQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:15.546649037 (60h55m46.736947611s) [noise=true , address=79.127.205.234:4619 , piece_id=EZWLZOEWF6FC625TPK4PE6PM2PJB37ILDVRZ6M4L4GWG5T54ESCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:16.052389924 (60h55m47.242688504s) [noise=true , address=45.91.116.14:4206 , piece_id=HOPWK6GRYV7JG3OUX2QMAJ2B75OVR4J36QO7KCFU5RQKE54UQSJQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:19.609570723 (60h55m50.799869297s) [noise=true , address=77.90.31.19:4871 , piece_id=46PZAK3YFMAK2XCAVBAYREWN4Q7Y6BYR7P5WHZBE5E5VKLIWY2UQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:26.137992383 (60h55m57.328290964s) [noise=true , address=79.127.201.215:4356 , piece_id=JRZJEAU3VMXU57T4G6ADAMKCH7M2TYSZUTEQZFTAITG2OZESTWCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:34.085410066 (60h56m5.27570864s) [noise=true , address=104.36.82.56:4238 , piece_id=WZPLGWBE4KING7ENAG7YHEWXR5EY56CUL2YXRLK275446YKXLSSA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:31:52.466630613 (60h56m23.656929184s) [noise=true , address=104.36.82.47:4549 , piece_id=QYEFJCAO2JL2V7GOXSYIKDI3KHXZOSOCAYDX36AAJMOFTTD5H2JQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:32:03.443345661 (60h56m34.633644233s) [noise=true , address=104.36.82.46:4490 , piece_id=MV72INCBFO3CDX5MMKYDKRVES2KGRPZMHLP77DJSALAAIKXUXO3Q, seg_pos={0 0}, upload=58403]: Done uploading piece. err: <nil> Jul 27 00:32:13.149943633 (60h56m44.34024222s) [noise=true , address=77.90.31.18:4821 , piece_id=VA32SCYMUL3WCDV7ONS4FTF6ZWRZKMSRUALP22V273QEUWOYD56A, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:32:13.150052881 (60h56m44.340351461s) [noise=true , address=104.36.82.21:4499 , piece_id=S2R2DQWWHTLLTWPKGAHCNWS6BLQXQLO2VWYBC2RF27WA7NABFBBA, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:1218y8DtJNTTu94dbqGhywmHR4s99zcz9issmMmpgwusr2KgZxc, address:104.36.82.21:4499): protocol: write tcp 139.59.144.22:35346->104.36.82.21:4499: use of closed network connection; write tcp 139.59.144.22:35346->104.36.82.21:4499: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35346->104.36.82.21:4499: use of closed network connection Jul 27 00:32:13.150145509 (60h56m44.340444089s) [noise=true , address=104.36.82.53:4132 , piece_id=2QDVBG4PLYCWEBOJJVIWULIXBVFXPYQGKSMTYVWZ4ZXAQ5VZCJYQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:12tJshEftJnivXzEquETFS1gMFbPbVH63hpLrxwGLMwLfixFVup, address:104.36.82.53:4132): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:32:13.150225277 (60h56m44.340523857s) [noise=true , address=77.90.31.15:4722 , piece_id=O7YQIFEWMUWHW2HZENCBREBWMHNB5E7C6UQDIPRGQ2HKGWSZXRCQ, seg_pos={0 0}, upload=58403]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: rpc: dial tcp 77.90.31.15:4722: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:13.150256167 (60h56m44.34055475s) [seg_pos={0 0}, upload=58403]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 27 00:32:13.256444579 (60h56m44.446743162s) [upload=58403]: Done uploading segments... Jul 27 00:32:13.256475069 (60h56m44.44677365s) [upload=58403]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.108889278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [X..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX.] duration: 3.769040076s FAIL (2025-07-27 00:32:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 3.769073559s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [XX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXXX] duration: 30.109127224s FAIL (2025-07-27 00:32:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.109150148s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......................X.X........................] duration: 852.632474ms FAIL (2025-07-27 00:31:51 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 852.725787ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX......XXX] duration: 30.11363061s FAIL (2025-07-27 00:31:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 5.601812308s) [upload=58405]: Starting upload Jul 27 00:31:14.411590961 (60h55m45.601889554s) [upload=58405]: Done starting upload Jul 27 00:31:14.411743878 (60h55m45.602042447s) [upload=58405]: Uploading segments... Jul 27 00:31:14.412065887 (60h55m45.602364762s) [upload=58405]: Got next segment. Inline: false Jul 27 00:31:14.412118939 (60h55m45.602417518s) [upload=58405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:14.412124854 (60h55m45.602423434s) [upload=58405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:14.524075014 (60h55m45.714373604s) [seg_pos={0 0}, upload=58405]: Begin upload segment... Jul 27 00:31:14.524144658 (60h55m45.714443248s) [seg_pos={0 0}, upload=58405]: Done begin upload segment. Jul 27 00:31:14.524174801 (60h55m45.714473395s) [noise=true , address=45.91.116.13:4154 , piece_id=IVELWLS37AHY2XJF7IL5AM5DGHI5NJLON35N5WDH2NKCHOWJMA2A, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.524191748 (60h55m45.714490328s) [noise=true , address=77.90.31.15:4698 , piece_id=LDKO2LZVSDBGAAHHRF24IHPRB6Q4BH5RDSKBLUOI7OMAUBTBSSZA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.524264906 (60h55m45.714563485s) [noise=true , address=45.91.116.11:4118 , piece_id=7MIETURSLQOUMMDQKSFFGCGS2YZXWTCOA5TQUSJSBKEPY6XBIWGQ, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.524630244 (60h55m45.714928825s) [noise=true , address=45.91.116.5:4297 , piece_id=JK5IG4DPFENRDLUBX7ZWQXNZ432Z43ERIQPDFINCPDLYRDWFMMIA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.524689013 (60h55m45.714987603s) [noise=true , address=77.90.31.16:4735 , piece_id=OJTXKDXRA65URC3I3TB7SCYWS3S25KXAO2CBHTW5T735UI3HQVLA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.524951993 (60h55m45.715250579s) [noise=true , address=77.90.31.10:4518 , piece_id=I26545F7RYP7DLYHBXGO6SA3TATUES7UY3WNDHLUJ4JSIEEG7MKA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.525200132 (60h55m45.715498723s) [noise=true , address=77.90.31.15:4699 , piece_id=SZDOKB6PDPMZR7QQRP6NGMSMELTW5UHV2ITJL5AATEW3RPLYRMKQ, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.525368445 (60h55m45.715667031s) [noise=true , address=104.36.82.13:4187 , piece_id=WDOOZPXQCGG44WZTSKPMV6GT2MCBELETYTD2CXTKNJJIYSNGEC7Q, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.525613478 (60h55m45.715912067s) [noise=true , address=45.91.116.12:4226 , piece_id=YUTHX3FE3WX66ZPR5UGOKIGTJAFRGIPRWLQZKWM6C7BOM4JMRPOA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.525746954 (60h55m45.716045536s) [noise=true , address=104.36.82.7:4909 , piece_id=VWH77XI77JLYWATD6NFOURMJPSRLCZNREIIA7K2L5SXYP4RFGRRA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.526044807 (60h55m45.716343397s) [noise=true , address=104.36.82.8:4970 , piece_id=FMAZDS4TVPTMVTKBO7GDJZ73AZUZBCSNFTKUVVEXH7LE2DHWPR7Q, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.526450669 (60h55m45.716749256s) [noise=true , address=104.36.82.10:4079 , piece_id=6PEX5TWQPOTK7MASMMX2CHS3SQLU2BDTD3V3NEBTLK4KFXN266QA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.526839900 (60h55m45.71713849s) [noise=true , address=45.91.116.10:4081 , piece_id=3LPTGLZVBU2SRKO5W35DE6FGSLRL7XY7IGKIY2UPXS4YVDRZE66Q, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.527230383 (60h55m45.717528987s) [noise=true , address=104.36.82.5:4838 , piece_id=6QX2ZQ6GOGDGLZTFKGB54WGNAFOKROZSKWCOXHIVOJTHQ7HGPIVQ, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.527598116 (60h55m45.71789669s) [noise=true , address=45.91.116.4:4940 , piece_id=GW5EB3LHHPCMHA3GZ2V3TNYLFQ4POCT5HLHXSZM7LSQYY2GA4NLA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:14.531794709 (60h55m45.72209331s) [noise=true , address=45.91.116.8:4045 , piece_id=C6W4BAVFX6X426HOPD7LODM67BRW2B66HZOCBSD7ZGGE3OUCZEVA, seg_pos={0 0}, upload=58405]: Uploading piece... Jul 27 00:31:15.131264031 (60h55m46.321562611s) [upload=58405]: Next returned nil segment Jul 27 00:31:15.131282600 (60h55m46.321581178s) [upload=58405]: Waiting for error group managing segments... Jul 27 00:31:15.267263194 (60h55m46.457561762s) [noise=true , address=45.91.116.13:4154 , piece_id=IVELWLS37AHY2XJF7IL5AM5DGHI5NJLON35N5WDH2NKCHOWJMA2A, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.273344760 (60h55m46.463643343s) [noise=true , address=45.91.116.5:4297 , piece_id=JK5IG4DPFENRDLUBX7ZWQXNZ432Z43ERIQPDFINCPDLYRDWFMMIA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.276860588 (60h55m46.46715918s) [noise=true , address=45.91.116.12:4226 , piece_id=YUTHX3FE3WX66ZPR5UGOKIGTJAFRGIPRWLQZKWM6C7BOM4JMRPOA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.280907815 (60h55m46.471206398s) [noise=true , address=77.90.31.10:4518 , piece_id=I26545F7RYP7DLYHBXGO6SA3TATUES7UY3WNDHLUJ4JSIEEG7MKA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.283751487 (60h55m46.474050058s) [noise=true , address=45.91.116.4:4940 , piece_id=GW5EB3LHHPCMHA3GZ2V3TNYLFQ4POCT5HLHXSZM7LSQYY2GA4NLA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.284902656 (60h55m46.475201239s) [noise=true , address=45.91.116.8:4045 , piece_id=C6W4BAVFX6X426HOPD7LODM67BRW2B66HZOCBSD7ZGGE3OUCZEVA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.294157285 (60h55m46.484455875s) [noise=true , address=45.91.116.11:4118 , piece_id=7MIETURSLQOUMMDQKSFFGCGS2YZXWTCOA5TQUSJSBKEPY6XBIWGQ, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.298594464 (60h55m46.488893045s) [noise=true , address=45.91.116.10:4081 , piece_id=3LPTGLZVBU2SRKO5W35DE6FGSLRL7XY7IGKIY2UPXS4YVDRZE66Q, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:15.304161978 (60h55m46.494460558s) [noise=true , address=77.90.31.16:4735 , piece_id=OJTXKDXRA65URC3I3TB7SCYWS3S25KXAO2CBHTW5T735UI3HQVLA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:42.891928578 (60h56m14.082227147s) [noise=true , address=104.36.82.5:4838 , piece_id=6QX2ZQ6GOGDGLZTFKGB54WGNAFOKROZSKWCOXHIVOJTHQ7HGPIVQ, seg_pos={0 0}, upload=58405]: Done uploading piece. err: <nil> Jul 27 00:31:44.139971765 (60h56m15.330270344s) [noise=true , address=104.36.82.8:4970 , piece_id=FMAZDS4TVPTMVTKBO7GDJZ73AZUZBCSNFTKUVVEXH7LE2DHWPR7Q, seg_pos={0 0}, upload=58405]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 139.59.144.22:44234->104.36.82.8:4970: use of closed network connection; write tcp 139.59.144.22:44234->104.36.82.8:4970: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44234->104.36.82.8:4970: use of closed network connection Jul 27 00:31:44.140042666 (60h56m15.330341245s) [noise=true , address=104.36.82.7:4909 , piece_id=VWH77XI77JLYWATD6NFOURMJPSRLCZNREIIA7K2L5SXYP4RFGRRA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:55284->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:55284->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55284->104.36.82.7:4909: use of closed network connection Jul 27 00:31:44.140099699 (60h56m15.330398278s) [noise=true , address=77.90.31.15:4698 , piece_id=LDKO2LZVSDBGAAHHRF24IHPRB6Q4BH5RDSKBLUOI7OMAUBTBSSZA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:44.140151317 (60h56m15.330449897s) [noise=true , address=77.90.31.15:4699 , piece_id=SZDOKB6PDPMZR7QQRP6NGMSMELTW5UHV2ITJL5AATEW3RPLYRMKQ, seg_pos={0 0}, upload=58405]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:44.140200736 (60h56m15.330499316s) [noise=true , address=104.36.82.13:4187 , piece_id=WDOOZPXQCGG44WZTSKPMV6GT2MCBELETYTD2CXTKNJJIYSNGEC7Q, seg_pos={0 0}, upload=58405]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 139.59.144.22:54730->104.36.82.13:4187: use of closed network connection; write tcp 139.59.144.22:54730->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54730->104.36.82.13:4187: use of closed network connection Jul 27 00:31:44.140370409 (60h56m15.330668989s) [noise=true , address=104.36.82.10:4079 , piece_id=6PEX5TWQPOTK7MASMMX2CHS3SQLU2BDTD3V3NEBTLK4KFXN266QA, seg_pos={0 0}, upload=58405]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: read tcp 139.59.144.22:51224->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:44.140387433 (60h56m15.330686003s) [seg_pos={0 0}, upload=58405]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:31:44.251207851 (60h56m15.441506434s) [upload=58405]: Done uploading segments... Jul 27 00:31:44.251242595 (60h56m15.441541184s) [upload=58405]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.113661565s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......X..X............X.X..X.X..............................] duration: 24.687018058s FAIL (2025-07-27 00:31:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.687053856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX...X....X.......XX...X....X...XX....X......XXXX..XX...X...] duration: 4.123389023s FAIL (2025-07-27 00:31:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.123418334s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XXX] duration: 30.109628788s FAIL (2025-07-27 00:31:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.10966477s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX......XX] duration: 30.119192853s FAIL (2025-07-27 00:31:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .36.82.3:4767): protocol: write tcp 139.59.144.22:37332->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:37332->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37332->104.36.82.3:4767: use of closed network connection Jul 27 00:30:32.595200986 (60h55m3.785499566s) [seg_pos={0 0}, upload=58384]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:30:32.698377352 (60h55m3.88867592s) [upload=58384]: Done uploading segments... Jul 27 00:30:32.698412954 (60h55m3.888711522s) [upload=58384]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 27 00:30:40.235071142 (60h55m11.425369726s) [upload=58395]: Starting upload Jul 27 00:30:40.235160465 (60h55m11.425459044s) [upload=58395]: Done starting upload Jul 27 00:30:40.235438545 (60h55m11.425737115s) [upload=58395]: Uploading segments... Jul 27 00:30:40.235489132 (60h55m11.425787753s) [upload=58395]: Got next segment. Inline: false Jul 27 00:30:40.235554504 (60h55m11.425853083s) [upload=58395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:30:40.235560204 (60h55m11.425858783s) [upload=58395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:30:40.346778808 (60h55m11.537077395s) [seg_pos={0 0}, upload=58395]: Begin upload segment... Jul 27 00:30:40.346874670 (60h55m11.53717326s) [seg_pos={0 0}, upload=58395]: Done begin upload segment. Jul 27 00:30:40.346921889 (60h55m11.537220482s) [noise=true , address=45.91.116.12:4227 , piece_id=B44OUGZ3W6MLERV7JUL5IRAJ5DK6TEP7KRN5EJYB7M2KYVR6WKIQ, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.346950998 (60h55m11.537249589s) [noise=true , address=45.91.116.14:4190 , piece_id=IAOYJGAXSRJC6I7OJ4GQ2K5M4DADFPAFYSP67RVIFJJF5N3VRGXA, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.347046196 (60h55m11.537344785s) [noise=true , address=45.91.116.9:4261 , piece_id=YIHGFU77BU6UVPM2IYXHR56KRF4NLPLXTHNOESB42YGKOMDVL63Q, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.347630024 (60h55m11.537928614s) [noise=true , address=77.90.31.17:4771 , piece_id=RW7IO2OYCNQG5TJ2RF5NUOVRXARAD4QXZIRG2OOOBCTG2ZDS64QA, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.347645546 (60h55m11.537944129s) [noise=true , address=45.91.116.11:4117 , piece_id=UDGDJ4BP7KR6YZ4BTNM5A76OBXYREAY6FJYVYNA3BOCPF35NLM5A, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.348151708 (60h55m11.538450295s) [noise=true , address=45.91.116.7:4011 , piece_id=MS3TRQLBGYF65DRXTICHUQ6NKOTQ7NB54I3QPKD4LYG5SN3KGYSQ, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.348661619 (60h55m11.538960208s) [noise=true , address=45.91.116.12:4225 , piece_id=UURPYPWMBMWEHYXF4JPNGTB6MZ3XVVOG2UAAWLA5QVWWR3BMX6TA, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.349271401 (60h55m11.539569983s) [noise=true , address=77.90.31.14:4662 , piece_id=7C3XCWRPWA56CN3JV724OXOPMRT25BJKKKM2VSVG4BH3YLPDMSAA, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.349687098 (60h55m11.539985671s) [noise=true , address=77.90.31.13:4628 , piece_id=TYUZHGXJOWTW6YLBHRI555HW3S3ZCLPRZA3ZOBYMXSZCEPGNHOZQ, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.351133464 (60h55m11.541432053s) [noise=true , address=104.36.82.13:4186 , piece_id=O3JP6DXFQEKY4V72JIOX6YC4EHB65DBNPOQQNPCOBTQVD4QFTXYA, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.351929461 (60h55m11.542228051s) [noise=true , address=45.91.116.7:4009 , piece_id=IWHMKC7FDVCTDXXOPDF2FR7UDBFYKLT5J4HGYRDMMQMQO5T3X5XA, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.352732772 (60h55m11.543031363s) [noise=true , address=104.36.82.4:4801 , piece_id=WFPESFUFGESYGQ22PVZUBUJTH3XFG57HHLPG7RQNKAPISUMX5MCQ, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.353438056 (60h55m11.543736646s) [noise=true , address=77.90.31.10:4519 , piece_id=EXCKP43AVVWA5X2BSXZTSGCCYNYYW2I4MG2XBQG5SQ3ZLID53REQ, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.353966529 (60h55m11.544265119s) [noise=true , address=104.36.82.4:4804 , piece_id=WVHSUFHBRYDH4M5JD2FWDJPNIBKRSC4VRQZ4FSRAKSHKO4PCLLOQ, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.359496138 (60h55m11.54979473s) [noise=true , address=45.91.116.4:4939 , piece_id=363QLCEXTO3V2CIHTW3OWUDZRU4LTTWMHF3BKRLYAOLW6MNHOVCQ, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.373619560 (60h55m11.563918145s) [noise=true , address=45.91.116.8:4048 , piece_id=HROJSH3EMQT5Y76NP3AIGLUSBG3Q6AFDHIDTRHTREQVO7MF6GWVA, seg_pos={0 0}, upload=58395]: Uploading piece... Jul 27 00:30:40.962086870 (60h55m12.152385449s) [upload=58395]: Next returned nil segment Jul 27 00:30:40.962110344 (60h55m12.152408927s) [upload=58395]: Waiting for error group managing segments... Jul 27 00:30:41.096657130 (60h55m12.28695572s) [noise=true , address=45.91.116.12:4227 , piece_id=B44OUGZ3W6MLERV7JUL5IRAJ5DK6TEP7KRN5EJYB7M2KYVR6WKIQ, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.097397659 (60h55m12.287696246s) [noise=true , address=45.91.116.11:4117 , piece_id=UDGDJ4BP7KR6YZ4BTNM5A76OBXYREAY6FJYVYNA3BOCPF35NLM5A, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.106465537 (60h55m12.296764116s) [noise=true , address=45.91.116.14:4190 , piece_id=IAOYJGAXSRJC6I7OJ4GQ2K5M4DADFPAFYSP67RVIFJJF5N3VRGXA, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.109562735 (60h55m12.299861301s) [noise=true , address=45.91.116.9:4261 , piece_id=YIHGFU77BU6UVPM2IYXHR56KRF4NLPLXTHNOESB42YGKOMDVL63Q, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.112841075 (60h55m12.303139647s) [noise=true , address=45.91.116.8:4048 , piece_id=HROJSH3EMQT5Y76NP3AIGLUSBG3Q6AFDHIDTRHTREQVO7MF6GWVA, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.114273777 (60h55m12.304572356s) [noise=true , address=45.91.116.4:4939 , piece_id=363QLCEXTO3V2CIHTW3OWUDZRU4LTTWMHF3BKRLYAOLW6MNHOVCQ, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.114734783 (60h55m12.305033358s) [noise=true , address=45.91.116.7:4009 , piece_id=IWHMKC7FDVCTDXXOPDF2FR7UDBFYKLT5J4HGYRDMMQMQO5T3X5XA, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.125906875 (60h55m12.316205454s) [noise=true , address=77.90.31.14:4662 , piece_id=7C3XCWRPWA56CN3JV724OXOPMRT25BJKKKM2VSVG4BH3YLPDMSAA, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.132748447 (60h55m12.32304703s) [noise=true , address=45.91.116.12:4225 , piece_id=UURPYPWMBMWEHYXF4JPNGTB6MZ3XVVOG2UAAWLA5QVWWR3BMX6TA, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.197751108 (60h55m12.388049689s) [noise=true , address=45.91.116.7:4011 , piece_id=MS3TRQLBGYF65DRXTICHUQ6NKOTQ7NB54I3QPKD4LYG5SN3KGYSQ, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:41.214738553 (60h55m12.405037118s) [noise=true , address=77.90.31.10:4519 , piece_id=EXCKP43AVVWA5X2BSXZTSGCCYNYYW2I4MG2XBQG5SQ3ZLID53REQ, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:30:42.528552008 (60h55m13.71885058s) [noise=true , address=77.90.31.13:4628 , piece_id=TYUZHGXJOWTW6YLBHRI555HW3S3ZCLPRZA3ZOBYMXSZCEPGNHOZQ, seg_pos={0 0}, upload=58395]: Done uploading piece. err: <nil> Jul 27 00:31:09.960886128 (60h55m41.151184693s) [noise=true , address=104.36.82.13:4186 , piece_id=O3JP6DXFQEKY4V72JIOX6YC4EHB65DBNPOQQNPCOBTQVD4QFTXYA, seg_pos={0 0}, upload=58395]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:31:09.960973844 (60h55m41.151272423s) [noise=true , address=104.36.82.4:4801 , piece_id=WFPESFUFGESYGQ22PVZUBUJTH3XFG57HHLPG7RQNKAPISUMX5MCQ, seg_pos={0 0}, upload=58395]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:47342->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:47342->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47342->104.36.82.4:4801: use of closed network connection Jul 27 00:31:09.961893517 (60h55m41.152192084s) [noise=true , address=77.90.31.17:4771 , piece_id=RW7IO2OYCNQG5TJ2RF5NUOVRXARAD4QXZIRG2OOOBCTG2ZDS64QA, seg_pos={0 0}, upload=58395]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:09.962033741 (60h55m41.152332309s) [noise=true , address=104.36.82.4:4804 , piece_id=WVHSUFHBRYDH4M5JD2FWDJPNIBKRSC4VRQZ4FSRAKSHKO4PCLLOQ, seg_pos={0 0}, upload=58395]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 139.59.144.22:49770->104.36.82.4:4804: use of closed network connection; write tcp 139.59.144.22:49770->104.36.82.4:4804: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49770->104.36.82.4:4804: use of closed network connection Jul 27 00:31:09.962053194 (60h55m41.15235178s) [seg_pos={0 0}, upload=58395]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:10.074116288 (60h55m41.26441486s) [upload=58395]: Done uploading segments... Jul 27 00:31:10.074141221 (60h55m41.26443979s) [upload=58395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.119218932s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX...X....X.......XX...X....X...XX....X......XXXX..XX...X..] duration: 14.562403229s FAIL (2025-07-27 00:31:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.562428483s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X......X..X............X.X..X.X.............................] duration: 36.613099735s FAIL (2025-07-27 00:31:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.613124357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [X......X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX..] duration: 24.282981046s FAIL (2025-07-27 00:31:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== enough pieces (needed at least 14 but got 11) Jul 27 00:29:00.689585417 (60h53m31.879884006s) [upload=58366]: Starting upload Jul 27 00:29:00.689711487 (60h53m31.880010076s) [upload=58366]: Done starting upload Jul 27 00:29:00.689827844 (60h53m31.880126413s) [upload=58366]: Uploading segments... Jul 27 00:29:00.690181170 (60h53m31.880479762s) [upload=58366]: Got next segment. Inline: false Jul 27 00:29:00.690229103 (60h53m31.880527683s) [upload=58366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:29:00.690260950 (60h53m31.880559532s) [upload=58366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:29:00.889613733 (60h53m32.079912324s) [seg_pos={0 0}, upload=58366]: Begin upload segment... Jul 27 00:29:00.889709456 (60h53m32.080008056s) [seg_pos={0 0}, upload=58366]: Done begin upload segment. Jul 27 00:29:00.889740914 (60h53m32.080039493s) [noise=true , address=104.36.82.54:4168 , piece_id=OYSFVH6PMJKIOGKCPNALNBC4CJSLS5VOXTIRU2RKVQMHEZFKW53Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.889765976 (60h53m32.080064565s) [noise=true , address=104.36.82.18:4372 , piece_id=VLZWYRYAYN5D5DVQOMTGH2OBB3EXH7MBQQH7G2J4NUBBTUA2TQJQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890190603 (60h53m32.080489186s) [noise=true , address=64.49.13.26:4953 , piece_id=45HUSMJT2ACULYB7HBSJN4BQ46A7AL2YULDBCQ7RNAFCT5XYG7YA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890398260 (60h53m32.08069685s) [noise=true , address=79.127.220.101:4048 , piece_id=AWSM7UBFOPJPS36YR4D2VIZPNKQ6SWB55JGMJD3N5FTYZU6P2GNA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890932670 (60h53m32.08123126s) [noise=true , address=79.127.219.38:4004 , piece_id=5B76EVW5MAC6C3UIF4KF2CLAC6H5FXJWNJMNUBWAEB2IEDMZSV3Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.891432852 (60h53m32.081731442s) [noise=true , address=77.90.31.15:4714 , piece_id=JGZP6JCFKKP5ZLH5LXFZYXOLYNZCO22WV4J6LVWZXWKVT33YOUQQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.891883419 (60h53m32.082182008s) [noise=true , address=104.36.82.52:4096 , piece_id=36CDC35VL6U76AKLOM2VPIPHAIOFSAMHM4N6N2VB3IBSZG3T7VBQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.892334968 (60h53m32.082633559s) [noise=true , address=104.36.82.53:4125 , piece_id=Z6HWBJIPC2YYSD77MV56AX6OPQNVBKQRKKSZXRXHQQEJUMU4YHBQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.892809584 (60h53m32.083108173s) [noise=true , address=104.36.82.46:4507 , piece_id=2A2ZY5OTQ22W4OIB2VRC2566TNOALVFZQG3BZADXF4NTE3HW2MDA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.893362394 (60h53m32.083660982s) [noise=true , address=79.127.205.234:4606 , piece_id=6K42L6U3I7I2KTHOGJCEVKAYEUMFKMZ3PGVBYHES4BLZE6XME77Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.893888553 (60h53m32.084187144s) [noise=true , address=104.36.82.55:4218 , piece_id=GSOV57XKLAA4VF7XDM24MWIKPFHAPYG2HSZW5ICJ5OUS6QFEOA7Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.898450833 (60h53m32.088749423s) [noise=true , address=79.127.205.233:4652 , piece_id=3QPSGB7R4MLI4JLYR4UCE5S3ZHPAFNZIPTIWFX32KUWTYCVD46DA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.898892310 (60h53m32.089190889s) [noise=true , address=64.49.13.138:4488 , piece_id=TIV6DBO3CG4A7GTA2X7X4IQSY4PYBDHJQYQRW3HP4WURECLIXAXA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899234508 (60h53m32.089533088s) [noise=true , address=77.90.31.21:4895 , piece_id=LGDGIV2T5BLRVYHDML6CG36HVMC6G5CLRT3ZNBSVILGPMDTI3Z3Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899570097 (60h53m32.089868677s) [noise=true , address=64.49.13.140:4508 , piece_id=I4O5C3NTZ2N5TNSEYY6EC6YX7ZQCH4U6OYXNOKMYJXAIZAVACZZA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899861702 (60h53m32.090160283s) [noise=true , address=79.127.205.246:4282 , piece_id=K35MZKPAQLGKTLAD7VBJQACHJVNEF3OLJXD3CNK5E5NB5V2AGQ7Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.900207088 (60h53m32.090505668s) [noise=true , address=77.90.31.17:4787 , piece_id=TJZ65ONGOVAXEGQRGFAPXLLZR4NMFEINHHZ42ZD3ZEF53GK5SVLQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.900265888 (60h53m32.09056448s) [noise=true , address=104.36.82.47:4524 , piece_id=YPUOOFOAWRCG5327UYJ4XI2IQWW5E5BPPRTYTYVTIFM7YIYFVE5Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:01.470565303 (60h53m32.660863889s) [upload=58366]: Next returned nil segment Jul 27 00:29:01.470591971 (60h53m32.66089056s) [upload=58366]: Waiting for error group managing segments... Jul 27 00:29:01.659112621 (60h53m32.849411187s) [noise=true , address=79.127.219.38:4004 , piece_id=5B76EVW5MAC6C3UIF4KF2CLAC6H5FXJWNJMNUBWAEB2IEDMZSV3Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:01.662680418 (60h53m32.852978991s) [noise=true , address=77.90.31.21:4895 , piece_id=LGDGIV2T5BLRVYHDML6CG36HVMC6G5CLRT3ZNBSVILGPMDTI3Z3Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:01.917707934 (60h53m33.108006543s) [noise=true , address=64.49.13.26:4953 , piece_id=45HUSMJT2ACULYB7HBSJN4BQ46A7AL2YULDBCQ7RNAFCT5XYG7YA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:29:02.159644479 (60h53m33.349943061s) [noise=true , address=79.127.205.234:4606 , piece_id=6K42L6U3I7I2KTHOGJCEVKAYEUMFKMZ3PGVBYHES4BLZE6XME77Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.162742332 (60h53m33.353040904s) [noise=true , address=64.49.13.140:4508 , piece_id=I4O5C3NTZ2N5TNSEYY6EC6YX7ZQCH4U6OYXNOKMYJXAIZAVACZZA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.262209880 (60h53m33.452508451s) [noise=true , address=79.127.205.246:4282 , piece_id=K35MZKPAQLGKTLAD7VBJQACHJVNEF3OLJXD3CNK5E5NB5V2AGQ7Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.333406053 (60h53m33.52370466s) [noise=true , address=79.127.205.233:4652 , piece_id=3QPSGB7R4MLI4JLYR4UCE5S3ZHPAFNZIPTIWFX32KUWTYCVD46DA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.463400638 (60h53m33.653699228s) [noise=true , address=64.49.13.138:4488 , piece_id=TIV6DBO3CG4A7GTA2X7X4IQSY4PYBDHJQYQRW3HP4WURECLIXAXA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:03.913090845 (60h53m35.103389423s) [noise=true , address=79.127.220.101:4048 , piece_id=AWSM7UBFOPJPS36YR4D2VIZPNKQ6SWB55JGMJD3N5FTYZU6P2GNA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:11.058700963 (60h53m42.248999533s) [noise=true , address=104.36.82.54:4168 , piece_id=OYSFVH6PMJKIOGKCPNALNBC4CJSLS5VOXTIRU2RKVQMHEZFKW53Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:16.734543840 (60h53m47.924842425s) [noise=true , address=104.36.82.55:4218 , piece_id=GSOV57XKLAA4VF7XDM24MWIKPFHAPYG2HSZW5ICJ5OUS6QFEOA7Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:26.763292381 (60h53m57.95359095s) [noise=true , address=104.36.82.47:4524 , piece_id=YPUOOFOAWRCG5327UYJ4XI2IQWW5E5BPPRTYTYVTIFM7YIYFVE5Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:28.809813989 (60h54m0.000112561s) [noise=true , address=104.36.82.53:4125 , piece_id=Z6HWBJIPC2YYSD77MV56AX6OPQNVBKQRKKSZXRXHQQEJUMU4YHBQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:46.323274193 (60h54m17.51357278s) [noise=true , address=104.36.82.46:4507 , piece_id=2A2ZY5OTQ22W4OIB2VRC2566TNOALVFZQG3BZADXF4NTE3HW2MDA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:58.757289763 (60h54m29.947588332s) [noise=true , address=104.36.82.52:4096 , piece_id=36CDC35VL6U76AKLOM2VPIPHAIOFSAMHM4N6N2VB3IBSZG3T7VBQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:58.757328980 (60h54m29.947627559s) [seg_pos={0 0}, upload=58366]: Segment reached optimal threshold of 14 pieces. Jul 27 00:29:58.757500868 (60h54m29.947799451s) [noise=true , address=104.36.82.18:4372 , piece_id=VLZWYRYAYN5D5DVQOMTGH2OBB3EXH7MBQQH7G2J4NUBBTUA2TQJQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T): protocol: expected piece hash; context canceled; EOF Jul 27 00:29:58.757555216 (60h54m29.947853786s) [noise=true , address=77.90.31.15:4714 , piece_id=JGZP6JCFKKP5ZLH5LXFZYXOLYNZCO22WV4J6LVWZXWKVT33YOUQQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Jul 27 00:29:58.757676520 (60h54m29.947975099s) [noise=true , address=77.90.31.17:4787 , piece_id=TJZ65ONGOVAXEGQRGFAPXLLZR4NMFEINHHZ42ZD3ZEF53GK5SVLQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: context canceled; context canceled Jul 27 00:29:58.757697014 (60h54m29.947995584s) [seg_pos={0 0}, upload=58366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:29:58.757771925 (60h54m29.948070504s) [upload=58366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:29:58.757777602 (60h54m29.948076232s) [upload=58366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:29:58.921609046 (60h54m30.111907625s) [upload=58366]: Done uploading segments... Jul 27 00:29:58.921630031 (60h54m30.111928611s) [upload=58366]: Upload finished. err: <nil> Duration: 24.283036909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......................X.X.......................] duration: 1.006028s FAIL (2025-07-27 00:30:39 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.006049734s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX......X] duration: 30.106091261s FAIL (2025-07-27 00:30:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== .14:4223 , piece_id=UPOEMFEWGZUEKOWJDH5LACLBW3HE6VA46YWN4IZP33OKQTFS6AMA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.974789887 (60h54m34.165088481s) [noise=true , address=104.36.82.12:4150 , piece_id=LSH36Y64K4QAMXR7PANIHL2JFJXB2QM55XGB37QLEQJ5FNR5QADA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.975214847 (60h54m34.16551344s) [noise=true , address=45.91.116.14:4189 , piece_id=RYO6TECDEHEVTDKVLCJXUJHVY7P2A6AAWFWPIGO4MUWCIKGRGSQA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.975635485 (60h54m34.165934075s) [noise=true , address=104.36.82.6:4876 , piece_id=DW2TW5T2STETOFHSQYDVEGSQSLLB3K334PRSV7LPG24MVVHA4TWA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.976030576 (60h54m34.166329184s) [noise=true , address=77.90.31.19:4842 , piece_id=TG553P2WFJZPJ6H5A3BHGY57CSX7C5EGTKPU4N7Z6TO4TEVM7KXA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.976410639 (60h54m34.16670923s) [noise=true , address=45.91.116.6:4974 , piece_id=B2LNV3XK5RD4NOOZMS4C6LYSXPUNJX4MG4HBJO5FH6RPZQ3IC4HQ, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.976795985 (60h54m34.167094575s) [noise=true , address=45.91.116.8:4047 , piece_id=NIEUO5EY76SUIFNOEXB73G4GM3CHP745EIIAIYRE3BEVWNCIZ4SA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.977532956 (60h54m34.167831551s) [noise=true , address=104.36.82.5:4839 , piece_id=BW4AM3AAHGGUG4GTBOURJQP4XH72MTDAYYYCCRI5DEEZBWWIWA3Q, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.978192846 (60h54m34.168491441s) [noise=true , address=77.90.31.11:4554 , piece_id=SWZ5YKSZFNSBYSE7TIJ534V7FY4H3BLGPI64LEA4C4FKSUSJ2O3Q, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.978688269 (60h54m34.168986861s) [noise=true , address=104.36.82.3:4767 , piece_id=PRPAURXRTNLUZZGI6DXUZ26VXRLYMWQTWY7XY4SPTQYGJZHA6MNA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.979246332 (60h54m34.169545032s) [noise=true , address=104.36.82.16:4294 , piece_id=6PGCDYG7R7JXBDRXIUBN6RI3KXFKHKRLACWJ7YWMPQR2LBSHGQEA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.979831512 (60h54m34.170130096s) [noise=true , address=104.36.82.2:4730 , piece_id=3JNM4DYG7EU7CNNPHSZYSLIOFBNCZCMEGGAOD2BRTYNQEE6CVH6Q, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.980904597 (60h54m34.171203191s) [noise=true , address=104.36.82.14:4222 , piece_id=JZ64QD3W7MTQBIRBERLEC22GWIQECNA4XS7DUPC3TVTINQRUR23A, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:02.981612795 (60h54m34.171911394s) [noise=true , address=104.36.82.9:4005 , piece_id=354GW2J6ZAEBEIXF6YDX2ZBJUAJ52MJMTTT72KCMPD3VDHRLJPXA, seg_pos={0 0}, upload=58384]: Uploading piece... Jul 27 00:30:03.655808735 (60h54m34.846107324s) [upload=58384]: Next returned nil segment Jul 27 00:30:03.655835045 (60h54m34.846133638s) [upload=58384]: Waiting for error group managing segments... Jul 27 00:30:03.786288703 (60h54m34.976587296s) [noise=true , address=45.91.116.13:4153 , piece_id=LS4W5SA3BVMWBFWBM3MSPER3V7Q7N6R76UKOCJILO6NTGW75YZHA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: <nil> Jul 27 00:30:03.793669880 (60h54m34.983968457s) [noise=true , address=45.91.116.8:4047 , piece_id=NIEUO5EY76SUIFNOEXB73G4GM3CHP745EIIAIYRE3BEVWNCIZ4SA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: <nil> Jul 27 00:30:03.794091942 (60h54m34.984390532s) [noise=true , address=45.91.116.6:4974 , piece_id=B2LNV3XK5RD4NOOZMS4C6LYSXPUNJX4MG4HBJO5FH6RPZQ3IC4HQ, seg_pos={0 0}, upload=58384]: Done uploading piece. err: <nil> Jul 27 00:30:03.817803558 (60h54m35.008102136s) [noise=true , address=45.91.116.14:4189 , piece_id=RYO6TECDEHEVTDKVLCJXUJHVY7P2A6AAWFWPIGO4MUWCIKGRGSQA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: <nil> Jul 27 00:30:05.147517915 (60h54m36.337816486s) [noise=true , address=77.90.31.19:4842 , piece_id=TG553P2WFJZPJ6H5A3BHGY57CSX7C5EGTKPU4N7Z6TO4TEVM7KXA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: <nil> Jul 27 00:30:05.188204563 (60h54m36.378503136s) [noise=true , address=77.90.31.13:4627 , piece_id=VNKGXNWXQFN52S6ORUVG3D5UE3HUOTYPPHUBVTPQCSY3H4WRU6EQ, seg_pos={0 0}, upload=58384]: Done uploading piece. err: <nil> Jul 27 00:30:26.700819920 (60h54m57.891118492s) [noise=true , address=104.36.82.16:4294 , piece_id=6PGCDYG7R7JXBDRXIUBN6RI3KXFKHKRLACWJ7YWMPQR2LBSHGQEA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: <nil> Jul 27 00:30:32.594788297 (60h55m3.785086882s) [noise=true , address=104.36.82.2:4730 , piece_id=3JNM4DYG7EU7CNNPHSZYSLIOFBNCZCMEGGAOD2BRTYNQEE6CVH6Q, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:30:32.594859885 (60h55m3.785158468s) [noise=true , address=104.36.82.14:4222 , piece_id=JZ64QD3W7MTQBIRBERLEC22GWIQECNA4XS7DUPC3TVTINQRUR23A, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 139.59.144.22:34654->104.36.82.14:4222: use of closed network connection; write tcp 139.59.144.22:34654->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34654->104.36.82.14:4222: use of closed network connection Jul 27 00:30:32.594875506 (60h55m3.785174085s) [noise=true , address=104.36.82.14:4223 , piece_id=UPOEMFEWGZUEKOWJDH5LACLBW3HE6VA46YWN4IZP33OKQTFS6AMA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 139.59.144.22:39740->104.36.82.14:4223: use of closed network connection; write tcp 139.59.144.22:39740->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39740->104.36.82.14:4223: use of closed network connection Jul 27 00:30:32.594958402 (60h55m3.785256983s) [noise=true , address=104.36.82.6:4876 , piece_id=DW2TW5T2STETOFHSQYDVEGSQSLLB3K334PRSV7LPG24MVVHA4TWA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 139.59.144.22:33956->104.36.82.6:4876: use of closed network connection; write tcp 139.59.144.22:33956->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33956->104.36.82.6:4876: use of closed network connection Jul 27 00:30:32.595003922 (60h55m3.785302491s) [noise=true , address=104.36.82.5:4839 , piece_id=BW4AM3AAHGGUG4GTBOURJQP4XH72MTDAYYYCCRI5DEEZBWWIWA3Q, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:53102->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:53102->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53102->104.36.82.5:4839: use of closed network connection Jul 27 00:30:32.595044162 (60h55m3.785342733s) [noise=true , address=104.36.82.12:4150 , piece_id=LSH36Y64K4QAMXR7PANIHL2JFJXB2QM55XGB37QLEQJ5FNR5QADA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 139.59.144.22:55252->104.36.82.12:4150: use of closed network connection; write tcp 139.59.144.22:55252->104.36.82.12:4150: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55252->104.36.82.12:4150: use of closed network connection Jul 27 00:30:32.595109001 (60h55m3.785407582s) [noise=true , address=77.90.31.11:4554 , piece_id=SWZ5YKSZFNSBYSE7TIJ534V7FY4H3BLGPI64LEA4C4FKSUSJ2O3Q, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:32.595119458 (60h55m3.785418093s) [noise=true , address=104.36.82.9:4005 , piece_id=354GW2J6ZAEBEIXF6YDX2ZBJUAJ52MJMTTT72KCMPD3VDHRLJPXA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 139.59.144.22:36098->104.36.82.9:4005: use of closed network connection; write tcp 139.59.144.22:36098->104.36.82.9:4005: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36098->104.36.82.9:4005: use of closed network connection Jul 27 00:30:32.595152774 (60h55m3.785451341s) [noise=true , address=104.36.82.3:4767 , piece_id=PRPAURXRTNLUZZGI6DXUZ26VXRLYMWQTWY7XY4SPTQYGJZHA6MNA, seg_pos={0 0}, upload=58384]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:37332->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:37332->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37332->104.36.82.3:4767: use of closed network connection Jul 27 00:30:32.595200986 (60h55m3.785499566s) [seg_pos={0 0}, upload=58384]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:30:32.698377352 (60h55m3.88867592s) [upload=58384]: Done uploading segments... Jul 27 00:30:32.698412954 (60h55m3.888711522s) [upload=58384]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.106126351s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X......X..X............X.X..X.X............................] duration: 52.554914368s FAIL (2025-07-27 00:30:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.554937342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.X......X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX.] duration: 28.764151519s FAIL (2025-07-27 00:30:29 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== enough pieces (needed at least 14 but got 11) Jul 27 00:29:00.689585417 (60h53m31.879884006s) [upload=58366]: Starting upload Jul 27 00:29:00.689711487 (60h53m31.880010076s) [upload=58366]: Done starting upload Jul 27 00:29:00.689827844 (60h53m31.880126413s) [upload=58366]: Uploading segments... Jul 27 00:29:00.690181170 (60h53m31.880479762s) [upload=58366]: Got next segment. Inline: false Jul 27 00:29:00.690229103 (60h53m31.880527683s) [upload=58366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:29:00.690260950 (60h53m31.880559532s) [upload=58366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:29:00.889613733 (60h53m32.079912324s) [seg_pos={0 0}, upload=58366]: Begin upload segment... Jul 27 00:29:00.889709456 (60h53m32.080008056s) [seg_pos={0 0}, upload=58366]: Done begin upload segment. Jul 27 00:29:00.889740914 (60h53m32.080039493s) [noise=true , address=104.36.82.54:4168 , piece_id=OYSFVH6PMJKIOGKCPNALNBC4CJSLS5VOXTIRU2RKVQMHEZFKW53Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.889765976 (60h53m32.080064565s) [noise=true , address=104.36.82.18:4372 , piece_id=VLZWYRYAYN5D5DVQOMTGH2OBB3EXH7MBQQH7G2J4NUBBTUA2TQJQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890190603 (60h53m32.080489186s) [noise=true , address=64.49.13.26:4953 , piece_id=45HUSMJT2ACULYB7HBSJN4BQ46A7AL2YULDBCQ7RNAFCT5XYG7YA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890398260 (60h53m32.08069685s) [noise=true , address=79.127.220.101:4048 , piece_id=AWSM7UBFOPJPS36YR4D2VIZPNKQ6SWB55JGMJD3N5FTYZU6P2GNA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890932670 (60h53m32.08123126s) [noise=true , address=79.127.219.38:4004 , piece_id=5B76EVW5MAC6C3UIF4KF2CLAC6H5FXJWNJMNUBWAEB2IEDMZSV3Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.891432852 (60h53m32.081731442s) [noise=true , address=77.90.31.15:4714 , piece_id=JGZP6JCFKKP5ZLH5LXFZYXOLYNZCO22WV4J6LVWZXWKVT33YOUQQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.891883419 (60h53m32.082182008s) [noise=true , address=104.36.82.52:4096 , piece_id=36CDC35VL6U76AKLOM2VPIPHAIOFSAMHM4N6N2VB3IBSZG3T7VBQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.892334968 (60h53m32.082633559s) [noise=true , address=104.36.82.53:4125 , piece_id=Z6HWBJIPC2YYSD77MV56AX6OPQNVBKQRKKSZXRXHQQEJUMU4YHBQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.892809584 (60h53m32.083108173s) [noise=true , address=104.36.82.46:4507 , piece_id=2A2ZY5OTQ22W4OIB2VRC2566TNOALVFZQG3BZADXF4NTE3HW2MDA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.893362394 (60h53m32.083660982s) [noise=true , address=79.127.205.234:4606 , piece_id=6K42L6U3I7I2KTHOGJCEVKAYEUMFKMZ3PGVBYHES4BLZE6XME77Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.893888553 (60h53m32.084187144s) [noise=true , address=104.36.82.55:4218 , piece_id=GSOV57XKLAA4VF7XDM24MWIKPFHAPYG2HSZW5ICJ5OUS6QFEOA7Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.898450833 (60h53m32.088749423s) [noise=true , address=79.127.205.233:4652 , piece_id=3QPSGB7R4MLI4JLYR4UCE5S3ZHPAFNZIPTIWFX32KUWTYCVD46DA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.898892310 (60h53m32.089190889s) [noise=true , address=64.49.13.138:4488 , piece_id=TIV6DBO3CG4A7GTA2X7X4IQSY4PYBDHJQYQRW3HP4WURECLIXAXA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899234508 (60h53m32.089533088s) [noise=true , address=77.90.31.21:4895 , piece_id=LGDGIV2T5BLRVYHDML6CG36HVMC6G5CLRT3ZNBSVILGPMDTI3Z3Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899570097 (60h53m32.089868677s) [noise=true , address=64.49.13.140:4508 , piece_id=I4O5C3NTZ2N5TNSEYY6EC6YX7ZQCH4U6OYXNOKMYJXAIZAVACZZA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899861702 (60h53m32.090160283s) [noise=true , address=79.127.205.246:4282 , piece_id=K35MZKPAQLGKTLAD7VBJQACHJVNEF3OLJXD3CNK5E5NB5V2AGQ7Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.900207088 (60h53m32.090505668s) [noise=true , address=77.90.31.17:4787 , piece_id=TJZ65ONGOVAXEGQRGFAPXLLZR4NMFEINHHZ42ZD3ZEF53GK5SVLQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.900265888 (60h53m32.09056448s) [noise=true , address=104.36.82.47:4524 , piece_id=YPUOOFOAWRCG5327UYJ4XI2IQWW5E5BPPRTYTYVTIFM7YIYFVE5Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:01.470565303 (60h53m32.660863889s) [upload=58366]: Next returned nil segment Jul 27 00:29:01.470591971 (60h53m32.66089056s) [upload=58366]: Waiting for error group managing segments... Jul 27 00:29:01.659112621 (60h53m32.849411187s) [noise=true , address=79.127.219.38:4004 , piece_id=5B76EVW5MAC6C3UIF4KF2CLAC6H5FXJWNJMNUBWAEB2IEDMZSV3Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:01.662680418 (60h53m32.852978991s) [noise=true , address=77.90.31.21:4895 , piece_id=LGDGIV2T5BLRVYHDML6CG36HVMC6G5CLRT3ZNBSVILGPMDTI3Z3Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:01.917707934 (60h53m33.108006543s) [noise=true , address=64.49.13.26:4953 , piece_id=45HUSMJT2ACULYB7HBSJN4BQ46A7AL2YULDBCQ7RNAFCT5XYG7YA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:29:02.159644479 (60h53m33.349943061s) [noise=true , address=79.127.205.234:4606 , piece_id=6K42L6U3I7I2KTHOGJCEVKAYEUMFKMZ3PGVBYHES4BLZE6XME77Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.162742332 (60h53m33.353040904s) [noise=true , address=64.49.13.140:4508 , piece_id=I4O5C3NTZ2N5TNSEYY6EC6YX7ZQCH4U6OYXNOKMYJXAIZAVACZZA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.262209880 (60h53m33.452508451s) [noise=true , address=79.127.205.246:4282 , piece_id=K35MZKPAQLGKTLAD7VBJQACHJVNEF3OLJXD3CNK5E5NB5V2AGQ7Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.333406053 (60h53m33.52370466s) [noise=true , address=79.127.205.233:4652 , piece_id=3QPSGB7R4MLI4JLYR4UCE5S3ZHPAFNZIPTIWFX32KUWTYCVD46DA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.463400638 (60h53m33.653699228s) [noise=true , address=64.49.13.138:4488 , piece_id=TIV6DBO3CG4A7GTA2X7X4IQSY4PYBDHJQYQRW3HP4WURECLIXAXA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:03.913090845 (60h53m35.103389423s) [noise=true , address=79.127.220.101:4048 , piece_id=AWSM7UBFOPJPS36YR4D2VIZPNKQ6SWB55JGMJD3N5FTYZU6P2GNA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:11.058700963 (60h53m42.248999533s) [noise=true , address=104.36.82.54:4168 , piece_id=OYSFVH6PMJKIOGKCPNALNBC4CJSLS5VOXTIRU2RKVQMHEZFKW53Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:16.734543840 (60h53m47.924842425s) [noise=true , address=104.36.82.55:4218 , piece_id=GSOV57XKLAA4VF7XDM24MWIKPFHAPYG2HSZW5ICJ5OUS6QFEOA7Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:26.763292381 (60h53m57.95359095s) [noise=true , address=104.36.82.47:4524 , piece_id=YPUOOFOAWRCG5327UYJ4XI2IQWW5E5BPPRTYTYVTIFM7YIYFVE5Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:28.809813989 (60h54m0.000112561s) [noise=true , address=104.36.82.53:4125 , piece_id=Z6HWBJIPC2YYSD77MV56AX6OPQNVBKQRKKSZXRXHQQEJUMU4YHBQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:46.323274193 (60h54m17.51357278s) [noise=true , address=104.36.82.46:4507 , piece_id=2A2ZY5OTQ22W4OIB2VRC2566TNOALVFZQG3BZADXF4NTE3HW2MDA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:58.757289763 (60h54m29.947588332s) [noise=true , address=104.36.82.52:4096 , piece_id=36CDC35VL6U76AKLOM2VPIPHAIOFSAMHM4N6N2VB3IBSZG3T7VBQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:58.757328980 (60h54m29.947627559s) [seg_pos={0 0}, upload=58366]: Segment reached optimal threshold of 14 pieces. Jul 27 00:29:58.757500868 (60h54m29.947799451s) [noise=true , address=104.36.82.18:4372 , piece_id=VLZWYRYAYN5D5DVQOMTGH2OBB3EXH7MBQQH7G2J4NUBBTUA2TQJQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T): protocol: expected piece hash; context canceled; EOF Jul 27 00:29:58.757555216 (60h54m29.947853786s) [noise=true , address=77.90.31.15:4714 , piece_id=JGZP6JCFKKP5ZLH5LXFZYXOLYNZCO22WV4J6LVWZXWKVT33YOUQQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Jul 27 00:29:58.757676520 (60h54m29.947975099s) [noise=true , address=77.90.31.17:4787 , piece_id=TJZ65ONGOVAXEGQRGFAPXLLZR4NMFEINHHZ42ZD3ZEF53GK5SVLQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: context canceled; context canceled Jul 27 00:29:58.757697014 (60h54m29.947995584s) [seg_pos={0 0}, upload=58366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:29:58.757771925 (60h54m29.948070504s) [upload=58366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:29:58.757777602 (60h54m29.948076232s) [upload=58366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:29:58.921609046 (60h54m30.111907625s) [upload=58366]: Done uploading segments... Jul 27 00:29:58.921630031 (60h54m30.111928611s) [upload=58366]: Upload finished. err: <nil> Duration: 28.764178606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.XX] duration: 30.108739741s FAIL (2025-07-27 00:30:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.109583547s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [X.X......X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..XX] duration: 1m0.005665594s FAIL (2025-07-27 00:30:00 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== enough pieces (needed at least 14 but got 11) Jul 27 00:29:00.689585417 (60h53m31.879884006s) [upload=58366]: Starting upload Jul 27 00:29:00.689711487 (60h53m31.880010076s) [upload=58366]: Done starting upload Jul 27 00:29:00.689827844 (60h53m31.880126413s) [upload=58366]: Uploading segments... Jul 27 00:29:00.690181170 (60h53m31.880479762s) [upload=58366]: Got next segment. Inline: false Jul 27 00:29:00.690229103 (60h53m31.880527683s) [upload=58366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:29:00.690260950 (60h53m31.880559532s) [upload=58366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:29:00.889613733 (60h53m32.079912324s) [seg_pos={0 0}, upload=58366]: Begin upload segment... Jul 27 00:29:00.889709456 (60h53m32.080008056s) [seg_pos={0 0}, upload=58366]: Done begin upload segment. Jul 27 00:29:00.889740914 (60h53m32.080039493s) [noise=true , address=104.36.82.54:4168 , piece_id=OYSFVH6PMJKIOGKCPNALNBC4CJSLS5VOXTIRU2RKVQMHEZFKW53Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.889765976 (60h53m32.080064565s) [noise=true , address=104.36.82.18:4372 , piece_id=VLZWYRYAYN5D5DVQOMTGH2OBB3EXH7MBQQH7G2J4NUBBTUA2TQJQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890190603 (60h53m32.080489186s) [noise=true , address=64.49.13.26:4953 , piece_id=45HUSMJT2ACULYB7HBSJN4BQ46A7AL2YULDBCQ7RNAFCT5XYG7YA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890398260 (60h53m32.08069685s) [noise=true , address=79.127.220.101:4048 , piece_id=AWSM7UBFOPJPS36YR4D2VIZPNKQ6SWB55JGMJD3N5FTYZU6P2GNA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.890932670 (60h53m32.08123126s) [noise=true , address=79.127.219.38:4004 , piece_id=5B76EVW5MAC6C3UIF4KF2CLAC6H5FXJWNJMNUBWAEB2IEDMZSV3Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.891432852 (60h53m32.081731442s) [noise=true , address=77.90.31.15:4714 , piece_id=JGZP6JCFKKP5ZLH5LXFZYXOLYNZCO22WV4J6LVWZXWKVT33YOUQQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.891883419 (60h53m32.082182008s) [noise=true , address=104.36.82.52:4096 , piece_id=36CDC35VL6U76AKLOM2VPIPHAIOFSAMHM4N6N2VB3IBSZG3T7VBQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.892334968 (60h53m32.082633559s) [noise=true , address=104.36.82.53:4125 , piece_id=Z6HWBJIPC2YYSD77MV56AX6OPQNVBKQRKKSZXRXHQQEJUMU4YHBQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.892809584 (60h53m32.083108173s) [noise=true , address=104.36.82.46:4507 , piece_id=2A2ZY5OTQ22W4OIB2VRC2566TNOALVFZQG3BZADXF4NTE3HW2MDA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.893362394 (60h53m32.083660982s) [noise=true , address=79.127.205.234:4606 , piece_id=6K42L6U3I7I2KTHOGJCEVKAYEUMFKMZ3PGVBYHES4BLZE6XME77Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.893888553 (60h53m32.084187144s) [noise=true , address=104.36.82.55:4218 , piece_id=GSOV57XKLAA4VF7XDM24MWIKPFHAPYG2HSZW5ICJ5OUS6QFEOA7Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.898450833 (60h53m32.088749423s) [noise=true , address=79.127.205.233:4652 , piece_id=3QPSGB7R4MLI4JLYR4UCE5S3ZHPAFNZIPTIWFX32KUWTYCVD46DA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.898892310 (60h53m32.089190889s) [noise=true , address=64.49.13.138:4488 , piece_id=TIV6DBO3CG4A7GTA2X7X4IQSY4PYBDHJQYQRW3HP4WURECLIXAXA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899234508 (60h53m32.089533088s) [noise=true , address=77.90.31.21:4895 , piece_id=LGDGIV2T5BLRVYHDML6CG36HVMC6G5CLRT3ZNBSVILGPMDTI3Z3Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899570097 (60h53m32.089868677s) [noise=true , address=64.49.13.140:4508 , piece_id=I4O5C3NTZ2N5TNSEYY6EC6YX7ZQCH4U6OYXNOKMYJXAIZAVACZZA, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.899861702 (60h53m32.090160283s) [noise=true , address=79.127.205.246:4282 , piece_id=K35MZKPAQLGKTLAD7VBJQACHJVNEF3OLJXD3CNK5E5NB5V2AGQ7Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.900207088 (60h53m32.090505668s) [noise=true , address=77.90.31.17:4787 , piece_id=TJZ65ONGOVAXEGQRGFAPXLLZR4NMFEINHHZ42ZD3ZEF53GK5SVLQ, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:00.900265888 (60h53m32.09056448s) [noise=true , address=104.36.82.47:4524 , piece_id=YPUOOFOAWRCG5327UYJ4XI2IQWW5E5BPPRTYTYVTIFM7YIYFVE5Q, seg_pos={0 0}, upload=58366]: Uploading piece... Jul 27 00:29:01.470565303 (60h53m32.660863889s) [upload=58366]: Next returned nil segment Jul 27 00:29:01.470591971 (60h53m32.66089056s) [upload=58366]: Waiting for error group managing segments... Jul 27 00:29:01.659112621 (60h53m32.849411187s) [noise=true , address=79.127.219.38:4004 , piece_id=5B76EVW5MAC6C3UIF4KF2CLAC6H5FXJWNJMNUBWAEB2IEDMZSV3Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:01.662680418 (60h53m32.852978991s) [noise=true , address=77.90.31.21:4895 , piece_id=LGDGIV2T5BLRVYHDML6CG36HVMC6G5CLRT3ZNBSVILGPMDTI3Z3Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:01.917707934 (60h53m33.108006543s) [noise=true , address=64.49.13.26:4953 , piece_id=45HUSMJT2ACULYB7HBSJN4BQ46A7AL2YULDBCQ7RNAFCT5XYG7YA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:29:02.159644479 (60h53m33.349943061s) [noise=true , address=79.127.205.234:4606 , piece_id=6K42L6U3I7I2KTHOGJCEVKAYEUMFKMZ3PGVBYHES4BLZE6XME77Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.162742332 (60h53m33.353040904s) [noise=true , address=64.49.13.140:4508 , piece_id=I4O5C3NTZ2N5TNSEYY6EC6YX7ZQCH4U6OYXNOKMYJXAIZAVACZZA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.262209880 (60h53m33.452508451s) [noise=true , address=79.127.205.246:4282 , piece_id=K35MZKPAQLGKTLAD7VBJQACHJVNEF3OLJXD3CNK5E5NB5V2AGQ7Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.333406053 (60h53m33.52370466s) [noise=true , address=79.127.205.233:4652 , piece_id=3QPSGB7R4MLI4JLYR4UCE5S3ZHPAFNZIPTIWFX32KUWTYCVD46DA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:02.463400638 (60h53m33.653699228s) [noise=true , address=64.49.13.138:4488 , piece_id=TIV6DBO3CG4A7GTA2X7X4IQSY4PYBDHJQYQRW3HP4WURECLIXAXA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:03.913090845 (60h53m35.103389423s) [noise=true , address=79.127.220.101:4048 , piece_id=AWSM7UBFOPJPS36YR4D2VIZPNKQ6SWB55JGMJD3N5FTYZU6P2GNA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:11.058700963 (60h53m42.248999533s) [noise=true , address=104.36.82.54:4168 , piece_id=OYSFVH6PMJKIOGKCPNALNBC4CJSLS5VOXTIRU2RKVQMHEZFKW53Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:16.734543840 (60h53m47.924842425s) [noise=true , address=104.36.82.55:4218 , piece_id=GSOV57XKLAA4VF7XDM24MWIKPFHAPYG2HSZW5ICJ5OUS6QFEOA7Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:26.763292381 (60h53m57.95359095s) [noise=true , address=104.36.82.47:4524 , piece_id=YPUOOFOAWRCG5327UYJ4XI2IQWW5E5BPPRTYTYVTIFM7YIYFVE5Q, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:28.809813989 (60h54m0.000112561s) [noise=true , address=104.36.82.53:4125 , piece_id=Z6HWBJIPC2YYSD77MV56AX6OPQNVBKQRKKSZXRXHQQEJUMU4YHBQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:46.323274193 (60h54m17.51357278s) [noise=true , address=104.36.82.46:4507 , piece_id=2A2ZY5OTQ22W4OIB2VRC2566TNOALVFZQG3BZADXF4NTE3HW2MDA, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:58.757289763 (60h54m29.947588332s) [noise=true , address=104.36.82.52:4096 , piece_id=36CDC35VL6U76AKLOM2VPIPHAIOFSAMHM4N6N2VB3IBSZG3T7VBQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: <nil> Jul 27 00:29:58.757328980 (60h54m29.947627559s) [seg_pos={0 0}, upload=58366]: Segment reached optimal threshold of 14 pieces. Jul 27 00:29:58.757500868 (60h54m29.947799451s) [noise=true , address=104.36.82.18:4372 , piece_id=VLZWYRYAYN5D5DVQOMTGH2OBB3EXH7MBQQH7G2J4NUBBTUA2TQJQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T): protocol: expected piece hash; context canceled; EOF Jul 27 00:29:58.757555216 (60h54m29.947853786s) [noise=true , address=77.90.31.15:4714 , piece_id=JGZP6JCFKKP5ZLH5LXFZYXOLYNZCO22WV4J6LVWZXWKVT33YOUQQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Jul 27 00:29:58.757676520 (60h54m29.947975099s) [noise=true , address=77.90.31.17:4787 , piece_id=TJZ65ONGOVAXEGQRGFAPXLLZR4NMFEINHHZ42ZD3ZEF53GK5SVLQ, seg_pos={0 0}, upload=58366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: context canceled; context canceled Jul 27 00:29:58.757697014 (60h54m29.947995584s) [seg_pos={0 0}, upload=58366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 27 00:29:58.757771925 (60h54m29.948070504s) [upload=58366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:29:58.757777602 (60h54m29.948076232s) [upload=58366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:29:58.921609046 (60h54m30.111907625s) [upload=58366]: Done uploading segments... Jul 27 00:29:58.921630031 (60h54m30.111928611s) [upload=58366]: Upload finished. err: <nil> Duration: 1m0.005716325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX...X....X.......XX...X....X...XX....X......XXXX..XX...X.] duration: 2.670943744s FAIL (2025-07-27 00:29:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.670975535s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXX..X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX......] duration: 3.706110703s FAIL (2025-07-27 00:29:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 985s) [upload=58299]: Done starting upload Jul 27 00:24:44.443132856 (60h49m15.633431428s) [upload=58299]: Uploading segments... Jul 27 00:24:44.443273839 (60h49m15.633572408s) [upload=58299]: Got next segment. Inline: false Jul 27 00:24:44.443321496 (60h49m15.633620065s) [upload=58299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:44.443327276 (60h49m15.633625856s) [upload=58299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:44.547877084 (60h49m15.738175653s) [seg_pos={0 0}, upload=58299]: Begin upload segment... Jul 27 00:24:44.547945724 (60h49m15.738244294s) [seg_pos={0 0}, upload=58299]: Done begin upload segment. Jul 27 00:24:44.547972750 (60h49m15.738271332s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.548528880 (60h49m15.738827468s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549066060 (60h49m15.739364648s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549244684 (60h49m15.739543274s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551195934 (60h49m15.741494523s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551427600 (60h49m15.741726191s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.552358440 (60h49m15.742657025s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553231576 (60h49m15.743530162s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553668942 (60h49m15.743967526s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554003042 (60h49m15.744301628s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554295592 (60h49m15.744594175s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554542412 (60h49m15.744840997s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554788066 (60h49m15.745086647s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555088744 (60h49m15.745387343s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555468081 (60h49m15.745766671s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.556726397 (60h49m15.747024987s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:45.159006343 (60h49m16.349304912s) [upload=58299]: Next returned nil segment Jul 27 00:24:45.159024162 (60h49m16.349322741s) [upload=58299]: Waiting for error group managing segments... Jul 27 00:24:45.298784708 (60h49m16.489083273s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301414157 (60h49m16.491713205s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301819516 (60h49m16.492118083s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.303997091 (60h49m16.494295675s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.318889685 (60h49m16.509188258s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.320045893 (60h49m16.510344461s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.373308676 (60h49m16.563607245s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.704643810 (60h49m17.894942379s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.706755114 (60h49m17.897053696s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.730072330 (60h49m17.920370896s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.800147718 (60h49m17.990446291s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:25:14.178560094 (60h49m45.368858673s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection Jul 27 00:25:14.178711600 (60h49m45.36901018s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection Jul 27 00:25:14.178795683 (60h49m45.369094262s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection Jul 27 00:25:14.178860310 (60h49m45.369158889s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection Jul 27 00:25:14.178910351 (60h49m45.369208932s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection Jul 27 00:25:14.178949836 (60h49m45.369248429s) [seg_pos={0 0}, upload=58299]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:14.281453751 (60h49m45.471752319s) [upload=58299]: Done uploading segments... Jul 27 00:25:14.281482189 (60h49m45.471780759s) [upload=58299]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 3.706151706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXXX..X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX.....] duration: 11.98232738s FAIL (2025-07-27 00:29:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 985s) [upload=58299]: Done starting upload Jul 27 00:24:44.443132856 (60h49m15.633431428s) [upload=58299]: Uploading segments... Jul 27 00:24:44.443273839 (60h49m15.633572408s) [upload=58299]: Got next segment. Inline: false Jul 27 00:24:44.443321496 (60h49m15.633620065s) [upload=58299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:44.443327276 (60h49m15.633625856s) [upload=58299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:44.547877084 (60h49m15.738175653s) [seg_pos={0 0}, upload=58299]: Begin upload segment... Jul 27 00:24:44.547945724 (60h49m15.738244294s) [seg_pos={0 0}, upload=58299]: Done begin upload segment. Jul 27 00:24:44.547972750 (60h49m15.738271332s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.548528880 (60h49m15.738827468s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549066060 (60h49m15.739364648s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549244684 (60h49m15.739543274s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551195934 (60h49m15.741494523s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551427600 (60h49m15.741726191s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.552358440 (60h49m15.742657025s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553231576 (60h49m15.743530162s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553668942 (60h49m15.743967526s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554003042 (60h49m15.744301628s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554295592 (60h49m15.744594175s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554542412 (60h49m15.744840997s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554788066 (60h49m15.745086647s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555088744 (60h49m15.745387343s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555468081 (60h49m15.745766671s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.556726397 (60h49m15.747024987s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:45.159006343 (60h49m16.349304912s) [upload=58299]: Next returned nil segment Jul 27 00:24:45.159024162 (60h49m16.349322741s) [upload=58299]: Waiting for error group managing segments... Jul 27 00:24:45.298784708 (60h49m16.489083273s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301414157 (60h49m16.491713205s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301819516 (60h49m16.492118083s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.303997091 (60h49m16.494295675s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.318889685 (60h49m16.509188258s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.320045893 (60h49m16.510344461s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.373308676 (60h49m16.563607245s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.704643810 (60h49m17.894942379s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.706755114 (60h49m17.897053696s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.730072330 (60h49m17.920370896s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.800147718 (60h49m17.990446291s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:25:14.178560094 (60h49m45.368858673s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection Jul 27 00:25:14.178711600 (60h49m45.36901018s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection Jul 27 00:25:14.178795683 (60h49m45.369094262s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection Jul 27 00:25:14.178860310 (60h49m45.369158889s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection Jul 27 00:25:14.178910351 (60h49m45.369208932s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection Jul 27 00:25:14.178949836 (60h49m45.369248429s) [seg_pos={0 0}, upload=58299]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:14.281453751 (60h49m45.471752319s) [upload=58299]: Done uploading segments... Jul 27 00:25:14.281482189 (60h49m45.471780759s) [upload=58299]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.982372732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......................X.X......................] duration: 907.375851ms FAIL (2025-07-27 00:29:29 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 907.411328ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.X] duration: 30.107465627s FAIL (2025-07-27 00:29:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.107499256s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X......X..X............X.X..X.X...........................] duration: 40.840477236s FAIL (2025-07-27 00:29:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.840498852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXX...X....X.......XX...X....X...XX....X......XXXX..XX...X] duration: 30.001173538s FAIL (2025-07-27 00:29:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001760193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.X.X......X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..X] duration: 1m0.109874084s FAIL (2025-07-27 00:29:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 58s) [upload=58352]: Got next segment. Inline: false Jul 27 00:28:00.436174563 (60h52m31.62647316s) [upload=58352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:00.436183218 (60h52m31.626481786s) [upload=58352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:00.636508308 (60h52m31.826806879s) [seg_pos={0 0}, upload=58352]: Begin upload segment... Jul 27 00:28:00.636581311 (60h52m31.82687988s) [seg_pos={0 0}, upload=58352]: Done begin upload segment. Jul 27 00:28:00.636625944 (60h52m31.826924533s) [noise=true , address=45.91.116.14:4206 , piece_id=EMO3WQQS2PZXBJ4QHPAV7HRRUDJNXYM7TMMOIZK6QQPPAU2LB3ZA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.636644944 (60h52m31.826943525s) [noise=true , address=104.36.82.47:4539 , piece_id=27HROETJXNXFIIYYF5PRUT2KE6RERHXOUFRQQWUTJR6KJHURHJOQ, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.637206548 (60h52m31.827505137s) [noise=true , address=77.90.31.13:4634 , piece_id=XU6AJ3RXPDZO45LKSQFS422COWFCOKLXCFJWZMMSZYDISQOJ5XRA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.637219379 (60h52m31.827517968s) [noise=true , address=79.127.205.245:4237 , piece_id=NBEIPF2CT25A5LRG4TIM5RETVZBCXWJGTRODHH2TZTOAKIE27NSA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.637634044 (60h52m31.827932635s) [noise=true , address=104.36.82.56:4233 , piece_id=LG3SHTV4O2FMRIGR57733DUWM3FDVHDY6TW3TMXQ4QUYDJSO5YAQ, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.637680262 (60h52m31.827978851s) [noise=true , address=104.36.82.46:4490 , piece_id=WDBGMZIBO656FPVOE3623SRMALYEL2YWM5VRGSQOY3IWOCOBGE4A, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.638047251 (60h52m31.82834584s) [noise=true , address=104.36.82.53:4153 , piece_id=NHDB76HWQFPLJNUHKSRAZ5MKJ5FSHP726SSBQ6M3WLLYS7ROWYSA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.638287747 (60h52m31.828586336s) [noise=true , address=64.49.13.138:4483 , piece_id=HKPMLYQETGZKAVE2IU6OIBLXJJA4RIXS7YRXT6RJ66GWY2CAQF3Q, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.638481728 (60h52m31.828780317s) [noise=true , address=79.127.205.246:4280 , piece_id=LBOH757WDSY267E5A3OX4FH6VNDUOJE4CHJIO4U3JUXRIOELMJ7Q, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.638738120 (60h52m31.82903671s) [noise=true , address=79.127.201.214:4289 , piece_id=7CU6Z5RNOG674M2XQ7GK7P4TX5HQR2ECIPUQDO6ZJBE5DY2YXM2Q, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.638847725 (60h52m31.829146315s) [noise=true , address=77.90.31.11:4565 , piece_id=ZMGFYYAVYPG2VVUUWYB7KP6WEXEJ4JXMMF2AL2RRKUZYXEPOB2UA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.639135335 (60h52m31.829433927s) [noise=true , address=104.36.82.52:4104 , piece_id=AL7UI5HIPD3NJZGA2DQ3WJLGHEKKPCSVFV4OMDQPLDTX4ILYWNIA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.639207127 (60h52m31.829505714s) [noise=true , address=77.90.31.21:4887 , piece_id=QTY2W4MYLIIVLEYQEEDRKDP77NQAJKTQPDB2HV45PWKREYA62ADQ, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.639534151 (60h52m31.829832739s) [noise=true , address=79.127.201.215:4347 , piece_id=UIWTMG2WHIN4M2TPTLCLT6E7M4FWDOFLH4OTVK5FNXHBUGKHDFQQ, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.639587777 (60h52m31.829886366s) [noise=true , address=77.90.31.19:4871 , piece_id=DWUQ6ESXP5YE52TFYY3EYPUCIUQ3KGNDEQNJXMJHLO7WTZ5VA4PA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.639906280 (60h52m31.830204869s) [noise=true , address=79.127.205.235:4675 , piece_id=LG5FCVGTLDHLTZYNVP3IQ2O6ZBQTPWOJKA7UE4U3DHNVZBNNEHHA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.639933415 (60h52m31.83023198s) [noise=true , address=79.127.205.233:4653 , piece_id=BLLM2BOPM5FXRZFLTF5Z5AJQG2IDPX6V5HC3MJAE2Q4WGZA7YYYQ, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:00.642246604 (60h52m31.832545193s) [noise=true , address=77.90.31.17:4798 , piece_id=BWTF2GURC4NMAPTA2MTIWEMFXGEKIBQEDKO2DFXFTNZEFLOH7ZGA, seg_pos={0 0}, upload=58352]: Uploading piece... Jul 27 00:28:01.018938958 (60h52m32.209237526s) [noise=true , address=77.90.31.19:4871 , piece_id=DWUQ6ESXP5YE52TFYY3EYPUCIUQ3KGNDEQNJXMJHLO7WTZ5VA4PA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:01.206839284 (60h52m32.397137864s) [upload=58352]: Next returned nil segment Jul 27 00:28:01.206860086 (60h52m32.397158671s) [upload=58352]: Waiting for error group managing segments... Jul 27 00:28:01.398286179 (60h52m32.58858476s) [noise=true , address=77.90.31.21:4887 , piece_id=QTY2W4MYLIIVLEYQEEDRKDP77NQAJKTQPDB2HV45PWKREYA62ADQ, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:01.941783504 (60h52m33.132082085s) [noise=true , address=79.127.205.245:4237 , piece_id=NBEIPF2CT25A5LRG4TIM5RETVZBCXWJGTRODHH2TZTOAKIE27NSA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:01.955697066 (60h52m33.145995635s) [noise=true , address=79.127.205.233:4653 , piece_id=BLLM2BOPM5FXRZFLTF5Z5AJQG2IDPX6V5HC3MJAE2Q4WGZA7YYYQ, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:02.017505866 (60h52m33.207804449s) [noise=true , address=79.127.205.235:4675 , piece_id=LG5FCVGTLDHLTZYNVP3IQ2O6ZBQTPWOJKA7UE4U3DHNVZBNNEHHA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:02.713450801 (60h52m33.903749367s) [noise=true , address=79.127.205.246:4280 , piece_id=LBOH757WDSY267E5A3OX4FH6VNDUOJE4CHJIO4U3JUXRIOELMJ7Q, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:04.952366358 (60h52m36.142664925s) [noise=true , address=79.127.201.214:4289 , piece_id=7CU6Z5RNOG674M2XQ7GK7P4TX5HQR2ECIPUQDO6ZJBE5DY2YXM2Q, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:06.123886919 (60h52m37.314185493s) [noise=true , address=45.91.116.14:4206 , piece_id=EMO3WQQS2PZXBJ4QHPAV7HRRUDJNXYM7TMMOIZK6QQPPAU2LB3ZA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:10.922940314 (60h52m42.113238892s) [noise=true , address=64.49.13.138:4483 , piece_id=HKPMLYQETGZKAVE2IU6OIBLXJJA4RIXS7YRXT6RJ66GWY2CAQF3Q, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:18.050549109 (60h52m49.240847695s) [noise=true , address=104.36.82.53:4153 , piece_id=NHDB76HWQFPLJNUHKSRAZ5MKJ5FSHP726SSBQ6M3WLLYS7ROWYSA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:23.168861862 (60h52m54.359160433s) [noise=true , address=79.127.201.215:4347 , piece_id=UIWTMG2WHIN4M2TPTLCLT6E7M4FWDOFLH4OTVK5FNXHBUGKHDFQQ, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:36.022473929 (60h53m7.212772496s) [noise=true , address=77.90.31.13:4634 , piece_id=XU6AJ3RXPDZO45LKSQFS422COWFCOKLXCFJWZMMSZYDISQOJ5XRA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:28:37.709425608 (60h53m8.899724181s) [noise=true , address=104.36.82.56:4233 , piece_id=LG3SHTV4O2FMRIGR57733DUWM3FDVHDY6TW3TMXQ4QUYDJSO5YAQ, seg_pos={0 0}, upload=58352]: Done uploading piece. err: <nil> Jul 27 00:29:00.162841940 (60h53m31.353140513s) [noise=true , address=104.36.82.46:4490 , piece_id=WDBGMZIBO656FPVOE3623SRMALYEL2YWM5VRGSQOY3IWOCOBGE4A, seg_pos={0 0}, upload=58352]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:29:00.162929838 (60h53m31.353228407s) [noise=true , address=104.36.82.52:4104 , piece_id=AL7UI5HIPD3NJZGA2DQ3WJLGHEKKPCSVFV4OMDQPLDTX4ILYWNIA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: ecclient: upload failed (node:12qmPpCmXAvuJZn2KR5Kr4nHYwKEzWvwqZ82nqNka5u1mKaZmXs, address:104.36.82.52:4104): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:29:00.163034490 (60h53m31.353333087s) [noise=true , address=104.36.82.47:4539 , piece_id=27HROETJXNXFIIYYF5PRUT2KE6RERHXOUFRQQWUTJR6KJHURHJOQ, seg_pos={0 0}, upload=58352]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:29:00.163209982 (60h53m31.353508568s) [noise=true , address=77.90.31.11:4565 , piece_id=ZMGFYYAVYPG2VVUUWYB7KP6WEXEJ4JXMMF2AL2RRKUZYXEPOB2UA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: read tcp 139.59.144.22:36012->77.90.31.11:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:00.163221069 (60h53m31.353519648s) [noise=true , address=77.90.31.17:4798 , piece_id=BWTF2GURC4NMAPTA2MTIWEMFXGEKIBQEDKO2DFXFTNZEFLOH7ZGA, seg_pos={0 0}, upload=58352]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: rpc: dial tcp 77.90.31.17:4798: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:00.163258850 (60h53m31.353557445s) [seg_pos={0 0}, upload=58352]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:29:00.270936290 (60h53m31.461234862s) [upload=58352]: Done uploading segments... Jul 27 00:29:00.270973191 (60h53m31.461271778s) [upload=58352]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.109912968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............X.......................X.X.....................] duration: 879.704971ms FAIL (2025-07-27 00:28:56 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 879.883736ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXX..X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX....] duration: 12.354049052s FAIL (2025-07-27 00:28:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 985s) [upload=58299]: Done starting upload Jul 27 00:24:44.443132856 (60h49m15.633431428s) [upload=58299]: Uploading segments... Jul 27 00:24:44.443273839 (60h49m15.633572408s) [upload=58299]: Got next segment. Inline: false Jul 27 00:24:44.443321496 (60h49m15.633620065s) [upload=58299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:44.443327276 (60h49m15.633625856s) [upload=58299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:44.547877084 (60h49m15.738175653s) [seg_pos={0 0}, upload=58299]: Begin upload segment... Jul 27 00:24:44.547945724 (60h49m15.738244294s) [seg_pos={0 0}, upload=58299]: Done begin upload segment. Jul 27 00:24:44.547972750 (60h49m15.738271332s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.548528880 (60h49m15.738827468s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549066060 (60h49m15.739364648s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549244684 (60h49m15.739543274s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551195934 (60h49m15.741494523s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551427600 (60h49m15.741726191s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.552358440 (60h49m15.742657025s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553231576 (60h49m15.743530162s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553668942 (60h49m15.743967526s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554003042 (60h49m15.744301628s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554295592 (60h49m15.744594175s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554542412 (60h49m15.744840997s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554788066 (60h49m15.745086647s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555088744 (60h49m15.745387343s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555468081 (60h49m15.745766671s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.556726397 (60h49m15.747024987s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:45.159006343 (60h49m16.349304912s) [upload=58299]: Next returned nil segment Jul 27 00:24:45.159024162 (60h49m16.349322741s) [upload=58299]: Waiting for error group managing segments... Jul 27 00:24:45.298784708 (60h49m16.489083273s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301414157 (60h49m16.491713205s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301819516 (60h49m16.492118083s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.303997091 (60h49m16.494295675s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.318889685 (60h49m16.509188258s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.320045893 (60h49m16.510344461s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.373308676 (60h49m16.563607245s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.704643810 (60h49m17.894942379s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.706755114 (60h49m17.897053696s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.730072330 (60h49m17.920370896s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.800147718 (60h49m17.990446291s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:25:14.178560094 (60h49m45.368858673s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection Jul 27 00:25:14.178711600 (60h49m45.36901018s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection Jul 27 00:25:14.178795683 (60h49m45.369094262s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection Jul 27 00:25:14.178860310 (60h49m45.369158889s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection Jul 27 00:25:14.178910351 (60h49m45.369208932s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection Jul 27 00:25:14.178949836 (60h49m45.369248429s) [seg_pos={0 0}, upload=58299]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:14.281453751 (60h49m45.471752319s) [upload=58299]: Done uploading segments... Jul 27 00:25:14.281482189 (60h49m45.471780759s) [upload=58299]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.354081277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XXXXX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X.] duration: 22.905858938s FAIL (2025-07-27 00:28:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 22.905889931s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X......X..X............X.X..X.X..........................] duration: 31.943573645s FAIL (2025-07-27 00:28:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.943607298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [X.X.X......X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX..] duration: 58.242989968s FAIL (2025-07-27 00:27:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pieces (needed at least 14 but got 12) Jul 27 00:24:50.599234662 (60h49m21.789533245s) [upload=58302]: Starting upload Jul 27 00:24:50.599359540 (60h49m21.789658126s) [upload=58302]: Done starting upload Jul 27 00:24:50.599635186 (60h49m21.789933772s) [upload=58302]: Uploading segments... Jul 27 00:24:50.600005576 (60h49m21.790304162s) [upload=58302]: Got next segment. Inline: false Jul 27 00:24:50.600055641 (60h49m21.790354233s) [upload=58302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:50.600076456 (60h49m21.790375045s) [upload=58302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:50.819274224 (60h49m22.009572806s) [seg_pos={0 0}, upload=58302]: Begin upload segment... Jul 27 00:24:50.819337404 (60h49m22.009635976s) [seg_pos={0 0}, upload=58302]: Done begin upload segment. Jul 27 00:24:50.819367042 (60h49m22.009665621s) [noise=true , address=64.49.13.138:4483 , piece_id=UZHHYELB2IE2NFUUMUESSYXRZHYYOWBGQOA4JNM63UAZUY45A2BQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.819827990 (60h49m22.01012657s) [noise=true , address=104.36.82.54:4160 , piece_id=AKNWWSK3VRPCLB25DNA7ZYNFJP3BVCTKOTT2Q6OPTHYVRAYDMAJQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.820147623 (60h49m22.010446202s) [noise=true , address=45.91.116.9:4277 , piece_id=GBLIKVZYAIDYOUJTRSCLPLWBUQAKTQVRGYM7WYK27MACV66QJXAQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.820469318 (60h49m22.010767896s) [noise=true , address=79.127.201.215:4359 , piece_id=SEZZREHWQQDR6CY4DTQHIOOH6KHJ7O3QVOT2ZA4X33IUP5JE3ZOA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.820638726 (60h49m22.010937311s) [noise=true , address=104.36.82.2:4761 , piece_id=6LWL22YKU763MQYFO5H5LV7PGITSEUR3FCQP6JSM5SWIHUBE6RHQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.822185360 (60h49m22.012483949s) [noise=true , address=77.90.31.12:4615 , piece_id=KMD7GPVBLYSRCJA2IR4VLOEBE3OW6WJD5FNQRDPH44CPFPONORPQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.822627825 (60h49m22.012926413s) [noise=true , address=77.90.31.21:4880 , piece_id=YFL3ZQI6GFMOZMJWBN22W7LAAIIDPZCLKQVEXIPUINJXACMJFVZQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823005538 (60h49m22.01330412s) [noise=true , address=77.90.31.10:4536 , piece_id=JITRX7B5UPX65MSL3RS6SFWV5PME6HB3KZ336CDYQT6HR5VRRQCA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823389556 (60h49m22.013688144s) [noise=true , address=45.91.116.14:4206 , piece_id=B6GCEKWGRM2LRRBNIXA67WWTXARBKAAFR6ZTUIKCM2ZFQIEWSP2A, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823769079 (60h49m22.014067664s) [noise=true , address=79.127.205.234:4612 , piece_id=YSN6AYDSQ5R5I6ABK67IU36JIC7D4FPZOEBVESQV36HP5NWFO5GA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823979128 (60h49m22.01427772s) [noise=true , address=77.90.31.17:4774 , piece_id=7ZNLKZ25VA5IPMAL5FSL5SD2EYWCNN4ZKTO6ZWZEBNZQJOIC7VIQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.824443132 (60h49m22.014741716s) [noise=true , address=104.36.82.46:4482 , piece_id=UK73FNOVZVKIQHVE3H3ZYQNE6FFZCTYUUZ3FWLPNIWA6A4UTAIGA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.824942945 (60h49m22.015241536s) [noise=true , address=79.127.205.235:4669 , piece_id=RYDTC7YMQFN2AB5XC4JIZ2NR2SUIQ5TKACEMZ5HEGQN4AQMVAOGA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.825438672 (60h49m22.015737262s) [noise=true , address=79.127.205.246:4272 , piece_id=5OGG5GPML56YQ4NJYT7ZF63MARSBRE4T7WCRLECYFJ6ESHYWOQQA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.826264934 (60h49m22.016563521s) [noise=true , address=104.36.82.53:4130 , piece_id=GKNS26UNPTA4LSRILQOKKKAMXFQ6S3BZZNKDYU6BS73TC3HTXMSA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.828217515 (60h49m22.018516106s) [noise=true , address=79.127.219.47:4048 , piece_id=QX64TJMUE7EPJSKKLJ7V2BINN5LC4HVZRTRJPWMUHTE2YDDWI3HA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.829255079 (60h49m22.019553671s) [noise=true , address=104.36.82.47:4550 , piece_id=4JZ6RAA5BR44WVNYDISFKCMQ6GFGIXELSTYM2A4ZWYJX23S5LPVQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.829662200 (60h49m22.019960797s) [noise=true , address=77.90.31.13:4629 , piece_id=4OW7J6HD4P4PCDSTXQ6ISA2KZTATWXQLO74SH4NKT3FBSS7MGGNQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:51.237567226 (60h49m22.427865805s) [noise=true , address=77.90.31.12:4615 , piece_id=KMD7GPVBLYSRCJA2IR4VLOEBE3OW6WJD5FNQRDPH44CPFPONORPQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:1xaory9AHtW4ra15wmL2CagPKKJU2u9keFo1ntBMy8AzjpkmwR, address:77.90.31.12:4615): protocol: storage node overloaded, request limit: 2000 Jul 27 00:24:51.401468209 (60h49m22.591766789s) [upload=58302]: Next returned nil segment Jul 27 00:24:51.401485998 (60h49m22.591784577s) [upload=58302]: Waiting for error group managing segments... Jul 27 00:24:51.500576390 (60h49m22.690874969s) [noise=true , address=64.49.13.138:4483 , piece_id=UZHHYELB2IE2NFUUMUESSYXRZHYYOWBGQOA4JNM63UAZUY45A2BQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:24:51.587222978 (60h49m22.777521559s) [noise=true , address=77.90.31.21:4880 , piece_id=YFL3ZQI6GFMOZMJWBN22W7LAAIIDPZCLKQVEXIPUINJXACMJFVZQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:51.623818351 (60h49m22.814116923s) [noise=true , address=45.91.116.9:4277 , piece_id=GBLIKVZYAIDYOUJTRSCLPLWBUQAKTQVRGYM7WYK27MACV66QJXAQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:52.128456999 (60h49m23.318755585s) [noise=true , address=79.127.205.234:4612 , piece_id=YSN6AYDSQ5R5I6ABK67IU36JIC7D4FPZOEBVESQV36HP5NWFO5GA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:52.174694765 (60h49m23.364993351s) [noise=true , address=79.127.205.246:4272 , piece_id=5OGG5GPML56YQ4NJYT7ZF63MARSBRE4T7WCRLECYFJ6ESHYWOQQA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:52.178763911 (60h49m23.369062491s) [noise=true , address=79.127.205.235:4669 , piece_id=RYDTC7YMQFN2AB5XC4JIZ2NR2SUIQ5TKACEMZ5HEGQN4AQMVAOGA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:53.691185821 (60h49m24.8814844s) [noise=true , address=79.127.219.47:4048 , piece_id=QX64TJMUE7EPJSKKLJ7V2BINN5LC4HVZRTRJPWMUHTE2YDDWI3HA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:08.078403534 (60h49m39.268702123s) [noise=true , address=45.91.116.14:4206 , piece_id=B6GCEKWGRM2LRRBNIXA67WWTXARBKAAFR6ZTUIKCM2ZFQIEWSP2A, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:12.241762454 (60h49m43.432061044s) [noise=true , address=104.36.82.47:4550 , piece_id=4JZ6RAA5BR44WVNYDISFKCMQ6GFGIXELSTYM2A4ZWYJX23S5LPVQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:15.704748885 (60h49m46.895047469s) [noise=true , address=104.36.82.54:4160 , piece_id=AKNWWSK3VRPCLB25DNA7ZYNFJP3BVCTKOTT2Q6OPTHYVRAYDMAJQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:25.802443185 (60h49m56.992741751s) [noise=true , address=104.36.82.53:4130 , piece_id=GKNS26UNPTA4LSRILQOKKKAMXFQ6S3BZZNKDYU6BS73TC3HTXMSA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:30.874812406 (60h50m2.065110974s) [noise=true , address=79.127.201.215:4359 , piece_id=SEZZREHWQQDR6CY4DTQHIOOH6KHJ7O3QVOT2ZA4X33IUP5JE3ZOA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:38.908662504 (60h50m10.09896109s) [noise=true , address=77.90.31.13:4629 , piece_id=4OW7J6HD4P4PCDSTXQ6ISA2KZTATWXQLO74SH4NKT3FBSS7MGGNQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:41.872780031 (60h50m13.063078604s) [noise=true , address=104.36.82.46:4482 , piece_id=UK73FNOVZVKIQHVE3H3ZYQNE6FFZCTYUUZ3FWLPNIWA6A4UTAIGA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:50.329691672 (60h50m21.519990244s) [noise=true , address=77.90.31.17:4774 , piece_id=7ZNLKZ25VA5IPMAL5FSL5SD2EYWCNN4ZKTO6ZWZEBNZQJOIC7VIQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: read tcp 139.59.144.22:42136->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:50.329782487 (60h50m21.520081052s) [noise=true , address=104.36.82.2:4761 , piece_id=6LWL22YKU763MQYFO5H5LV7PGITSEUR3FCQP6JSM5SWIHUBE6RHQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:12CXHkaBwi15pBUoRiZHi6JP5rBDu1iVpWT1WFL3dCmk8LRRxqw, address:104.36.82.2:4761): protocol: write tcp 139.59.144.22:48586->104.36.82.2:4761: use of closed network connection; context deadline exceeded Jul 27 00:25:50.329847837 (60h50m21.520146418s) [noise=true , address=77.90.31.10:4536 , piece_id=JITRX7B5UPX65MSL3RS6SFWV5PME6HB3KZ336CDYQT6HR5VRRQCA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:50.329890513 (60h50m21.520189093s) [seg_pos={0 0}, upload=58302]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 27 00:25:50.431329926 (60h50m21.621628506s) [upload=58302]: Done uploading segments... Jul 27 00:25:50.431357302 (60h50m21.621655882s) [upload=58302]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 58.243037492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [..XXXXX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.X] duration: 30.12420474s FAIL (2025-07-27 00:27:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.124250015s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......................X.X....................] duration: 894.155895ms FAIL (2025-07-27 00:27:42 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 894.188924ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....X......X..X............X.X..X.X.........................] duration: 40.60687383s FAIL (2025-07-27 00:27:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.606897804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX...X....X.......XX...X....X...XX....X......XXXX..XX...] duration: 2.642416283s FAIL (2025-07-27 00:27:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.642446867s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXX..X.X.X.XX.XXXXX..XX.XX..XXXX.X...........X.XXXX.XX...] duration: 2.375953294s FAIL (2025-07-27 00:27:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 985s) [upload=58299]: Done starting upload Jul 27 00:24:44.443132856 (60h49m15.633431428s) [upload=58299]: Uploading segments... Jul 27 00:24:44.443273839 (60h49m15.633572408s) [upload=58299]: Got next segment. Inline: false Jul 27 00:24:44.443321496 (60h49m15.633620065s) [upload=58299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:44.443327276 (60h49m15.633625856s) [upload=58299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:44.547877084 (60h49m15.738175653s) [seg_pos={0 0}, upload=58299]: Begin upload segment... Jul 27 00:24:44.547945724 (60h49m15.738244294s) [seg_pos={0 0}, upload=58299]: Done begin upload segment. Jul 27 00:24:44.547972750 (60h49m15.738271332s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.548528880 (60h49m15.738827468s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549066060 (60h49m15.739364648s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.549244684 (60h49m15.739543274s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551195934 (60h49m15.741494523s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.551427600 (60h49m15.741726191s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.552358440 (60h49m15.742657025s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553231576 (60h49m15.743530162s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.553668942 (60h49m15.743967526s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554003042 (60h49m15.744301628s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554295592 (60h49m15.744594175s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554542412 (60h49m15.744840997s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.554788066 (60h49m15.745086647s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555088744 (60h49m15.745387343s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.555468081 (60h49m15.745766671s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:44.556726397 (60h49m15.747024987s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:24:45.159006343 (60h49m16.349304912s) [upload=58299]: Next returned nil segment Jul 27 00:24:45.159024162 (60h49m16.349322741s) [upload=58299]: Waiting for error group managing segments... Jul 27 00:24:45.298784708 (60h49m16.489083273s) [noise=true , address=77.90.31.15:4698 , piece_id=C5U5WFM6BTG5HB4H2HXSEIKKR2X477KE5C5KOZXGN2NLW34T5TWQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301414157 (60h49m16.491713205s) [noise=true , address=45.91.116.5:4297 , piece_id=KVGOSJ63FR6BYR7XPM6ALIBJ5CHYTWZJTFE5FK7O4CRF5MPU2EFQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.301819516 (60h49m16.492118083s) [noise=true , address=45.91.116.12:4226 , piece_id=KC3H5QTP6GGMUG2AQQC67B5WW6P7VT2F2SFGMTPI3FBEWX4KN2SA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.303997091 (60h49m16.494295675s) [noise=true , address=77.90.31.18:4805 , piece_id=XZNTQV7RHRMB32S24LSNTA32HZ77VEWHP22P7JPRCHFTXTHUNVRQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.318889685 (60h49m16.509188258s) [noise=true , address=45.91.116.8:4047 , piece_id=T26CFKJVNK53LKLU2HGFD3PE72UJIFU5UHKC7SNXOM6AIPRQLWBA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.320045893 (60h49m16.510344461s) [noise=true , address=45.91.116.6:4973 , piece_id=J4KV7KVWHQVKYAQWHJLHUKSDJRVLSUSVLDRO3VWUBHMH4H5V5TSQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:45.373308676 (60h49m16.563607245s) [noise=true , address=45.91.116.7:4011 , piece_id=2Z6IOJ2LTXAEIU5JVKKVKT4JTIASW777XV2BZ426DOFAA4QIM2CQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.704643810 (60h49m17.894942379s) [noise=true , address=77.90.31.19:4841 , piece_id=R4PD3AFBG3BE3P4DQYUHQME2JPCTJEAE6CWZJWXKNCCGWEXBH45Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.706755114 (60h49m17.897053696s) [noise=true , address=77.90.31.12:4592 , piece_id=JWIWO3DZLOFVEUDZ4LNYDPWO3R5DNRAELHXWKWVNZNXFTLPNHAVA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.730072330 (60h49m17.920370896s) [noise=true , address=77.90.31.12:4590 , piece_id=BCYMXUCT4EBDNTRBSW2DCCR25A4CG3R6WNMXWX4HRV2UFRTURDAQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:46.800147718 (60h49m17.990446291s) [noise=true , address=77.90.31.13:4627 , piece_id=VNIEO5N6WMU52RUWAPNARSARZPHBBBXSOJLDGD466XIP5573BESQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:25:14.178560094 (60h49m45.368858673s) [noise=true , address=104.36.82.13:4185 , piece_id=TIDUTSFIT4TPEIELQAESV5KRFKG7ERQXB54FQJMKK3SPSCXJSU4Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:40828->104.36.82.13:4185: use of closed network connection Jul 27 00:25:14.178711600 (60h49m45.36901018s) [noise=true , address=104.36.82.16:4294 , piece_id=G2PEPM7SXN4EB4CNR33PD6V5ZPXCHRFDV63FS6RG7I7DAIXV4GGA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49496->104.36.82.16:4294: use of closed network connection Jul 27 00:25:14.178795683 (60h49m45.369094262s) [noise=true , address=104.36.82.2:4731 , piece_id=RWZSBPD5TORF7TUPALWUC5QVGHT7Y5O4JHG2U5KXTWNFY2PGGKKQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:52990->104.36.82.2:4731: use of closed network connection Jul 27 00:25:14.178860310 (60h49m45.369158889s) [noise=true , address=104.36.82.4:4801 , piece_id=4IP6ST6N3H6GJZMMYNWH3NNBNIY4MO4C7LOL7AXOZDTUJWAP46MQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58364->104.36.82.4:4801: use of closed network connection Jul 27 00:25:14.178910351 (60h49m45.369208932s) [noise=true , address=104.36.82.11:4113 , piece_id=PEEY4XUPGWXH7ONJJUQUAB77574LKVVLVKJ2VZQXK4BNOLOKNK3Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60456->104.36.82.11:4113: use of closed network connection Jul 27 00:25:14.178949836 (60h49m45.369248429s) [seg_pos={0 0}, upload=58299]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:14.281453751 (60h49m45.471752319s) [upload=58299]: Done uploading segments... Jul 27 00:25:14.281482189 (60h49m45.471780759s) [upload=58299]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.37599475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [X..XXXXX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX.] duration: 3.396276745s FAIL (2025-07-27 00:27:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.396384905s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XX.X.X......X..X...X.X.X..X..XXX....XX......X..X..XX.XX.XXX.] duration: 57.706377825s FAIL (2025-07-27 00:26:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pieces (needed at least 14 but got 12) Jul 27 00:24:50.599234662 (60h49m21.789533245s) [upload=58302]: Starting upload Jul 27 00:24:50.599359540 (60h49m21.789658126s) [upload=58302]: Done starting upload Jul 27 00:24:50.599635186 (60h49m21.789933772s) [upload=58302]: Uploading segments... Jul 27 00:24:50.600005576 (60h49m21.790304162s) [upload=58302]: Got next segment. Inline: false Jul 27 00:24:50.600055641 (60h49m21.790354233s) [upload=58302]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:50.600076456 (60h49m21.790375045s) [upload=58302]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:50.819274224 (60h49m22.009572806s) [seg_pos={0 0}, upload=58302]: Begin upload segment... Jul 27 00:24:50.819337404 (60h49m22.009635976s) [seg_pos={0 0}, upload=58302]: Done begin upload segment. Jul 27 00:24:50.819367042 (60h49m22.009665621s) [noise=true , address=64.49.13.138:4483 , piece_id=UZHHYELB2IE2NFUUMUESSYXRZHYYOWBGQOA4JNM63UAZUY45A2BQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.819827990 (60h49m22.01012657s) [noise=true , address=104.36.82.54:4160 , piece_id=AKNWWSK3VRPCLB25DNA7ZYNFJP3BVCTKOTT2Q6OPTHYVRAYDMAJQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.820147623 (60h49m22.010446202s) [noise=true , address=45.91.116.9:4277 , piece_id=GBLIKVZYAIDYOUJTRSCLPLWBUQAKTQVRGYM7WYK27MACV66QJXAQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.820469318 (60h49m22.010767896s) [noise=true , address=79.127.201.215:4359 , piece_id=SEZZREHWQQDR6CY4DTQHIOOH6KHJ7O3QVOT2ZA4X33IUP5JE3ZOA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.820638726 (60h49m22.010937311s) [noise=true , address=104.36.82.2:4761 , piece_id=6LWL22YKU763MQYFO5H5LV7PGITSEUR3FCQP6JSM5SWIHUBE6RHQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.822185360 (60h49m22.012483949s) [noise=true , address=77.90.31.12:4615 , piece_id=KMD7GPVBLYSRCJA2IR4VLOEBE3OW6WJD5FNQRDPH44CPFPONORPQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.822627825 (60h49m22.012926413s) [noise=true , address=77.90.31.21:4880 , piece_id=YFL3ZQI6GFMOZMJWBN22W7LAAIIDPZCLKQVEXIPUINJXACMJFVZQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823005538 (60h49m22.01330412s) [noise=true , address=77.90.31.10:4536 , piece_id=JITRX7B5UPX65MSL3RS6SFWV5PME6HB3KZ336CDYQT6HR5VRRQCA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823389556 (60h49m22.013688144s) [noise=true , address=45.91.116.14:4206 , piece_id=B6GCEKWGRM2LRRBNIXA67WWTXARBKAAFR6ZTUIKCM2ZFQIEWSP2A, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823769079 (60h49m22.014067664s) [noise=true , address=79.127.205.234:4612 , piece_id=YSN6AYDSQ5R5I6ABK67IU36JIC7D4FPZOEBVESQV36HP5NWFO5GA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.823979128 (60h49m22.01427772s) [noise=true , address=77.90.31.17:4774 , piece_id=7ZNLKZ25VA5IPMAL5FSL5SD2EYWCNN4ZKTO6ZWZEBNZQJOIC7VIQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.824443132 (60h49m22.014741716s) [noise=true , address=104.36.82.46:4482 , piece_id=UK73FNOVZVKIQHVE3H3ZYQNE6FFZCTYUUZ3FWLPNIWA6A4UTAIGA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.824942945 (60h49m22.015241536s) [noise=true , address=79.127.205.235:4669 , piece_id=RYDTC7YMQFN2AB5XC4JIZ2NR2SUIQ5TKACEMZ5HEGQN4AQMVAOGA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.825438672 (60h49m22.015737262s) [noise=true , address=79.127.205.246:4272 , piece_id=5OGG5GPML56YQ4NJYT7ZF63MARSBRE4T7WCRLECYFJ6ESHYWOQQA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.826264934 (60h49m22.016563521s) [noise=true , address=104.36.82.53:4130 , piece_id=GKNS26UNPTA4LSRILQOKKKAMXFQ6S3BZZNKDYU6BS73TC3HTXMSA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.828217515 (60h49m22.018516106s) [noise=true , address=79.127.219.47:4048 , piece_id=QX64TJMUE7EPJSKKLJ7V2BINN5LC4HVZRTRJPWMUHTE2YDDWI3HA, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.829255079 (60h49m22.019553671s) [noise=true , address=104.36.82.47:4550 , piece_id=4JZ6RAA5BR44WVNYDISFKCMQ6GFGIXELSTYM2A4ZWYJX23S5LPVQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:50.829662200 (60h49m22.019960797s) [noise=true , address=77.90.31.13:4629 , piece_id=4OW7J6HD4P4PCDSTXQ6ISA2KZTATWXQLO74SH4NKT3FBSS7MGGNQ, seg_pos={0 0}, upload=58302]: Uploading piece... Jul 27 00:24:51.237567226 (60h49m22.427865805s) [noise=true , address=77.90.31.12:4615 , piece_id=KMD7GPVBLYSRCJA2IR4VLOEBE3OW6WJD5FNQRDPH44CPFPONORPQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:1xaory9AHtW4ra15wmL2CagPKKJU2u9keFo1ntBMy8AzjpkmwR, address:77.90.31.12:4615): protocol: storage node overloaded, request limit: 2000 Jul 27 00:24:51.401468209 (60h49m22.591766789s) [upload=58302]: Next returned nil segment Jul 27 00:24:51.401485998 (60h49m22.591784577s) [upload=58302]: Waiting for error group managing segments... Jul 27 00:24:51.500576390 (60h49m22.690874969s) [noise=true , address=64.49.13.138:4483 , piece_id=UZHHYELB2IE2NFUUMUESSYXRZHYYOWBGQOA4JNM63UAZUY45A2BQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:24:51.587222978 (60h49m22.777521559s) [noise=true , address=77.90.31.21:4880 , piece_id=YFL3ZQI6GFMOZMJWBN22W7LAAIIDPZCLKQVEXIPUINJXACMJFVZQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:51.623818351 (60h49m22.814116923s) [noise=true , address=45.91.116.9:4277 , piece_id=GBLIKVZYAIDYOUJTRSCLPLWBUQAKTQVRGYM7WYK27MACV66QJXAQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:52.128456999 (60h49m23.318755585s) [noise=true , address=79.127.205.234:4612 , piece_id=YSN6AYDSQ5R5I6ABK67IU36JIC7D4FPZOEBVESQV36HP5NWFO5GA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:52.174694765 (60h49m23.364993351s) [noise=true , address=79.127.205.246:4272 , piece_id=5OGG5GPML56YQ4NJYT7ZF63MARSBRE4T7WCRLECYFJ6ESHYWOQQA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:52.178763911 (60h49m23.369062491s) [noise=true , address=79.127.205.235:4669 , piece_id=RYDTC7YMQFN2AB5XC4JIZ2NR2SUIQ5TKACEMZ5HEGQN4AQMVAOGA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:24:53.691185821 (60h49m24.8814844s) [noise=true , address=79.127.219.47:4048 , piece_id=QX64TJMUE7EPJSKKLJ7V2BINN5LC4HVZRTRJPWMUHTE2YDDWI3HA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:08.078403534 (60h49m39.268702123s) [noise=true , address=45.91.116.14:4206 , piece_id=B6GCEKWGRM2LRRBNIXA67WWTXARBKAAFR6ZTUIKCM2ZFQIEWSP2A, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:12.241762454 (60h49m43.432061044s) [noise=true , address=104.36.82.47:4550 , piece_id=4JZ6RAA5BR44WVNYDISFKCMQ6GFGIXELSTYM2A4ZWYJX23S5LPVQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:15.704748885 (60h49m46.895047469s) [noise=true , address=104.36.82.54:4160 , piece_id=AKNWWSK3VRPCLB25DNA7ZYNFJP3BVCTKOTT2Q6OPTHYVRAYDMAJQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:25.802443185 (60h49m56.992741751s) [noise=true , address=104.36.82.53:4130 , piece_id=GKNS26UNPTA4LSRILQOKKKAMXFQ6S3BZZNKDYU6BS73TC3HTXMSA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:30.874812406 (60h50m2.065110974s) [noise=true , address=79.127.201.215:4359 , piece_id=SEZZREHWQQDR6CY4DTQHIOOH6KHJ7O3QVOT2ZA4X33IUP5JE3ZOA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:38.908662504 (60h50m10.09896109s) [noise=true , address=77.90.31.13:4629 , piece_id=4OW7J6HD4P4PCDSTXQ6ISA2KZTATWXQLO74SH4NKT3FBSS7MGGNQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:41.872780031 (60h50m13.063078604s) [noise=true , address=104.36.82.46:4482 , piece_id=UK73FNOVZVKIQHVE3H3ZYQNE6FFZCTYUUZ3FWLPNIWA6A4UTAIGA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: <nil> Jul 27 00:25:50.329691672 (60h50m21.519990244s) [noise=true , address=77.90.31.17:4774 , piece_id=7ZNLKZ25VA5IPMAL5FSL5SD2EYWCNN4ZKTO6ZWZEBNZQJOIC7VIQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: read tcp 139.59.144.22:42136->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:50.329782487 (60h50m21.520081052s) [noise=true , address=104.36.82.2:4761 , piece_id=6LWL22YKU763MQYFO5H5LV7PGITSEUR3FCQP6JSM5SWIHUBE6RHQ, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:12CXHkaBwi15pBUoRiZHi6JP5rBDu1iVpWT1WFL3dCmk8LRRxqw, address:104.36.82.2:4761): protocol: write tcp 139.59.144.22:48586->104.36.82.2:4761: use of closed network connection; context deadline exceeded Jul 27 00:25:50.329847837 (60h50m21.520146418s) [noise=true , address=77.90.31.10:4536 , piece_id=JITRX7B5UPX65MSL3RS6SFWV5PME6HB3KZ336CDYQT6HR5VRRQCA, seg_pos={0 0}, upload=58302]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:50.329890513 (60h50m21.520189093s) [seg_pos={0 0}, upload=58302]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 27 00:25:50.431329926 (60h50m21.621628506s) [upload=58302]: Done uploading segments... Jul 27 00:25:50.431357302 (60h50m21.621655882s) [upload=58302]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 57.706410979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [XX..XXXXX..XXXXX..XXXXX..XXXX.XX..X....XX....X.X...X.XX..XXX] duration: 30.118565031s FAIL (2025-07-27 00:26:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.118614385s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X....X......X..X............X.X..X.X........................] duration: 50.990110334s FAIL (2025-07-27 00:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.990132184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......................X.X...................] duration: 1.196692414s FAIL (2025-07-27 00:26:29 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.196726936s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX...X....X.......XX...X....X...XX....X......XXXX..XX..] duration: 4.94680285s FAIL (2025-07-27 00:26:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.946828156s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....XXX...X....X.......XX...X....X...XX....X......XXXX..XX.] duration: 19.185773901s FAIL (2025-07-27 00:26:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.185797077s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)