Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.434864963s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:57:54 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.434893145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X..........X...X..........................] duration: 2.192976901s trace: https://jaeger.dev.storj.io/trace/6bb9e3317e182100 FAIL (2025-11-08 17:57:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.193006835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X...........X.............................] duration: 945.421645ms trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:57:38 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 945.460323ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....X........X......................] duration: 943.188338ms trace: https://jaeger.dev.storj.io/trace/197ffda43a2a97c2 FAIL (2025-11-08 17:57:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 943.22019ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 2.303748819s trace: https://jaeger.dev.storj.io/trace/3ada5398ef16e7a5 FAIL (2025-11-08 17:57:33 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.303782427s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X..........X...X.........................] duration: 1.725213709s trace: https://jaeger.dev.storj.io/trace/6bb9e3317e182100 FAIL (2025-11-08 17:57:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.725255506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.242835464s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:57:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.242868136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X...........X............................] duration: 1.157654874s trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:57:16 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.157683216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................X....X........X.....................] duration: 1.100085546s trace: https://jaeger.dev.storj.io/trace/197ffda43a2a97c2 FAIL (2025-11-08 17:57:12 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.100116794s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 9 out of 60 is failed (threshold=3) [...................X......X.X.X.XX..XX.X....................] duration: 294.277794ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/282d694e86fcb700 FAIL (2025-11-08 17:57:10 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 294.310778ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 2.420445212s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:57:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.420470959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 2.072147162s trace: https://jaeger.dev.storj.io/trace/3ada5398ef16e7a5 FAIL (2025-11-08 17:57:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.072179218s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.709279169s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:56:51 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.70930615s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 9 out of 60 is failed (threshold=3) [....................X......X.X.X.XX..XX.X...................] duration: 292.48332ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/282d694e86fcb700 FAIL (2025-11-08 17:56:25 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 292.515694ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X..........X...X........................] duration: 1.672787417s trace: https://jaeger.dev.storj.io/trace/6bb9e3317e182100 FAIL (2025-11-08 17:56:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.672886228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....X........X....................] duration: 884.16694ms trace: https://jaeger.dev.storj.io/trace/197ffda43a2a97c2 FAIL (2025-11-08 17:56:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 884.430467ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 2.313219821s trace: https://jaeger.dev.storj.io/trace/3ada5398ef16e7a5 FAIL (2025-11-08 17:56:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.313247037s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 2.600593566s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:56:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.600629844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X...........X...........................] duration: 1.099799734s trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:56:08 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.099834325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 2.660818814s trace: https://jaeger.dev.storj.io/trace/3ada5398ef16e7a5 FAIL (2025-11-08 17:56:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.660851138s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.304407238s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:55:45 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.304433665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 2.316498999s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:55:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.316526278s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 9 out of 60 is failed (threshold=3) [.....................X......X.X.X.XX..XX.X..................] duration: 306.491115ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/282d694e86fcb700 FAIL (2025-11-08 17:55:23 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 306.519941ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.317361338s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:55:19 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.317405459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X..........X...X.......................] duration: 1.65438424s trace: https://jaeger.dev.storj.io/trace/6bb9e3317e182100 FAIL (2025-11-08 17:55:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.654406422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....X........X...................] duration: 913.282609ms trace: https://jaeger.dev.storj.io/trace/197ffda43a2a97c2 FAIL (2025-11-08 17:55:16 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 913.307614ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X........X..................] duration: 916.545367ms trace: https://jaeger.dev.storj.io/trace/197ffda43a2a97c2 FAIL (2025-11-08 17:55:01 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 916.60739ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.637081265s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:54:56 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.637120428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...........X..........................] duration: 841.350456ms trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:54:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 841.377849ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 2.124224196s trace: https://jaeger.dev.storj.io/trace/3ada5398ef16e7a5 FAIL (2025-11-08 17:54:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.124250559s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.472940909s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:54:38 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.473037815s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 9 out of 60 is failed (threshold=3) [......................X......X.X.X.XX..XX.X.................] duration: 292.728286ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/282d694e86fcb700 FAIL (2025-11-08 17:54:33 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 292.762278ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X..........X...X......................] duration: 1.674005923s trace: https://jaeger.dev.storj.io/trace/6bb9e3317e182100 FAIL (2025-11-08 17:54:29 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.674025313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X...........X.........................] duration: 864.521333ms trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:54:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 864.546717ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 2.315785671s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:54:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.315816537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X....X........X.................] duration: 941.105935ms trace: https://jaeger.dev.storj.io/trace/197ffda43a2a97c2 FAIL (2025-11-08 17:54:23 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 941.139706ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 2.199785633s trace: https://jaeger.dev.storj.io/trace/3ada5398ef16e7a5 FAIL (2025-11-08 17:54:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.199809493s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X..........X...X.....................] duration: 1.687515138s trace: https://jaeger.dev.storj.io/trace/6bb9e3317e182100 FAIL (2025-11-08 17:54:11 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.687625976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 2.553418925s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.553452936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...........X........................] duration: 904.416437ms trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:53:52 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 904.52386ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 2.281470686s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:53:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.281497224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....X........X................] duration: 1.017143282s trace: https://jaeger.dev.storj.io/trace/197ffda43a2a97c2 FAIL (2025-11-08 17:53:49 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.017162312s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.650066152s trace: https://jaeger.dev.storj.io/trace/667af7ecec748423 FAIL (2025-11-08 17:53:35 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.650098956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 2.293605009s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:53:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.29362962s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 9 out of 60 is failed (threshold=3) [.......................X......X.X.X.XX..XX.X................] duration: 281.267412ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/282d694e86fcb700 FAIL (2025-11-08 17:53:27 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 281.314553ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 2.59319583s trace: https://jaeger.dev.storj.io/trace/3ada5398ef16e7a5 FAIL (2025-11-08 17:53:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.593232751s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X...........X.......................] duration: 919.282477ms trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:53:16 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 919.306486ms core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 9 out of 60 is failed (threshold=3) [........................X......X.X.X.XX..XX.X...............] duration: 271.830508ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/282d694e86fcb700 FAIL (2025-11-08 17:53:06 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 271.86276ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 2.611817226s trace: https://jaeger.dev.storj.io/trace/69a842eacf594af2 FAIL (2025-11-08 17:53:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 8 17:40:10.804199015 (221h59m13.505791471s) [upload=195222]: Starting upload Nov 8 17:40:10.804309456 (221h59m13.505901906s) [upload=195222]: Done starting upload Nov 8 17:40:10.804406258 (221h59m13.505998698s) [upload=195222]: Uploading segments... Nov 8 17:40:10.804661289 (221h59m13.506253725s) [upload=195222]: Got next segment. Inline: false Nov 8 17:40:10.804719099 (221h59m13.506311551s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 17:40:10.804725438 (221h59m13.506317891s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 17:40:10.949670396 (221h59m13.651262854s) [seg_pos={0 0}, upload=195222]: Begin upload segment... Nov 8 17:40:10.949746707 (221h59m13.651339142s) [seg_pos={0 0}, upload=195222]: Done begin upload segment. Nov 8 17:40:10.949790013 (221h59m13.651382473s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950253370 (221h59m13.65184583s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950807476 (221h59m13.652399944s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.950980887 (221h59m13.652573364s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951299826 (221h59m13.652892277s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951606289 (221h59m13.653198938s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.951724371 (221h59m13.653316822s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952145494 (221h59m13.653737946s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952213692 (221h59m13.653806149s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.952860843 (221h59m13.654453304s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.953456202 (221h59m13.655048664s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.954613710 (221h59m13.656206169s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.955958107 (221h59m13.657550571s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.956631592 (221h59m13.658224053s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.958408615 (221h59m13.660001073s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:10.960102983 (221h59m13.661695424s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Uploading piece... Nov 8 17:40:11.554414981 (221h59m14.256007458s) [upload=195222]: Next returned nil segment Nov 8 17:40:11.554435392 (221h59m14.256027861s) [upload=195222]: Waiting for error group managing segments... Nov 8 17:40:11.692643728 (221h59m14.394236176s) [noise=true , address=45.91.116.7:4009 , piece_id=ODWUQB5ECHFVLCNT7ITGFAVWLCSM6LBIVZFY5SKBO4PRSUTMOMZQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.694922910 (221h59m14.396515359s) [noise=true , address=145.79.227.27:4690 , piece_id=AN2BZBRPGNKZJD5QULYN6NQVBA5MAUPNJDPKXRGNJ5JGIE6IWB6Q, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.696239693 (221h59m14.397832128s) [noise=true , address=77.90.31.11:4555 , piece_id=7CLWZCMGVXWS36F2CDJSVLWQRF3FSTRSUGOR7UVAXE6HUJS6V3UQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.704279553 (221h59m14.405872002s) [noise=true , address=145.79.227.20:4437 , piece_id=VSIHFVU3Q7SHJHMUZIVJLHZGOVSXTWH3NAIX4J4ZD65NVVG5V7XQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708695617 (221h59m14.410288057s) [noise=true , address=145.79.227.8:4329 , piece_id=TQPKCTCJY3ELZWXGYPL4SZK2RSRQCACPYBUTID32RHHMOTPMPDAA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.708860281 (221h59m14.410452732s) [noise=true , address=145.79.227.30:4763 , piece_id=POFXWZ7WXWFPMDWOWYRZKL67GHGMU5EZZN43HK5TLIAQ55QVWM4A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.709384760 (221h59m14.410977212s) [noise=true , address=145.79.227.31:4799 , piece_id=J2VT3EXBVJ6MKB2BJWXCAFFXEVM4252KDWEU3OVN3ZQAM4WVZTOA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710418945 (221h59m14.412011393s) [noise=true , address=77.90.31.15:4699 , piece_id=QR3Y5CPZUMRTPJMRPNHVCSMB36GX535CPSAIOARS5IBUODCE5LVA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.710576422 (221h59m14.412168861s) [noise=true , address=145.79.227.2:4114 , piece_id=4MDCN7P2JUQ7S62EJPJ7HDWDJPBWDNAHS4CS6QXJPKHNMDX3BN7A, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713641307 (221h59m14.415233755s) [noise=true , address=45.91.116.12:4226 , piece_id=NSVBQ4V2ATLOSMHQKJEVS5YO63QHGEK2L2WCA5JXCUX3XDNWLZPQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.713802367 (221h59m14.415394815s) [noise=true , address=145.79.227.14:4838 , piece_id=SVTJ5R62XOK4ZKYBOZKWTEHUIB4FZFN7BI7S6C23VRBO6T2MTUAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.715278378 (221h59m14.416870828s) [noise=true , address=45.91.116.7:4010 , piece_id=YDUKICF5PG5BRCEITK2OOI2KYIHZPL6CMQ22SDQ3OC3O3YYIHQGA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718089883 (221h59m14.419682337s) [noise=true , address=45.91.116.12:4227 , piece_id=CHHGEF6WCH5IHGAP4WLHGKFRL7NAHLB5WDCLGXOPBJUWTXYON2FA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: <nil> Nov 8 17:40:11.718111323 (221h59m14.419703784s) [seg_pos={0 0}, upload=195222]: Segment reached optimal threshold of 13 pieces. Nov 8 17:40:11.718414960 (221h59m14.420007418s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718423176 (221h59m14.420015623s) [noise=true , address=145.79.227.13:4803 , piece_id=TIKME7HJKZH3AJQUND4UTFJFHCFBAH7PYGUEBBL4LS2RCD2XSGTQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718551606 (221h59m14.420144063s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718564756 (221h59m14.420157212s) [noise=true , address=145.79.227.22:4510 , piece_id=AMMRIE4NKVKUHOFDKXSWT7JR4DJ75ECODZI7767BVW2ZOBMDNRAQ, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718608200 (221h59m14.420200645s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Nov 8 17:40:11.718617070 (221h59m14.420209524s) [noise=true , address=77.90.31.14:4663 , piece_id=QEGZOKM3V3FDFHDWECT633YDLXRCP3G3NSPRL772UTBBQWTWJ5UA, seg_pos={0 0}, upload=195222]: Long tail cancellation due to optimal threshold Nov 8 17:40:11.718641555 (221h59m14.420234006s) [seg_pos={0 0}, upload=195222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 8 17:40:11.718729173 (221h59m14.420321626s) [upload=195222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 8 17:40:11.718736122 (221h59m14.420328576s) [upload=195222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 8 17:44:48.551903088 (222h3m51.253495548s) [upload=195222]: Done uploading segments... Nov 8 17:44:48.551995784 (222h3m51.253588254s) [upload=195222]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.611859222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...........X......................] duration: 1.108402517s trace: https://jaeger.dev.storj.io/trace/507a0a6af90e29da FAIL (2025-11-08 17:52:57 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.108445351s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)