Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X...X..X..X..........................................] duration:
2.503283481s
FAIL (2025-10-06 23:48:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.503312114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.772092089s
FAIL (2025-10-06 23:48:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.772166191s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X......XX...X............................................] duration:
2.620083281s
FAIL (2025-10-06 23:47:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.620114882s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [X.......X...X..X..X.........................................] duration:
2.380257231s
FAIL (2025-10-06 23:47:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.380288186s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
2.844937703s
FAIL (2025-10-06 23:47:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.844978268s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [X.......X..X...X.........X..................................] duration:
3.351214305s
FAIL (2025-10-06 23:47:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.351242376s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X......XX...X...........................................] duration:
2.701779755s
FAIL (2025-10-06 23:47:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.701812362s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X......XX...X..........................................] duration:
2.667308798s
FAIL (2025-10-06 23:46:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.667337757s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.X.......X..X...X.........X.................................] duration:
3.540365488s
FAIL (2025-10-06 23:46:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.540389725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X.......X...X..X..X........................................] duration:
2.301381411s
FAIL (2025-10-06 23:46:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.301410811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.568514238s
FAIL (2025-10-06 23:46:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.568542236s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
2.867791233s
FAIL (2025-10-06 23:46:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.868198469s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X.......X...X..X..X.......................................] duration:
2.517454524s
FAIL (2025-10-06 23:46:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.517487446s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
2.804560161s
FAIL (2025-10-06 23:45:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.804603879s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X......XX...X.........................................] duration:
2.786327196s
FAIL (2025-10-06 23:45:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.78635892s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..X.......X..X...X.........X................................] duration:
2.939393697s
FAIL (2025-10-06 23:45:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.939410443s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...X.......X..X...X.........X...............................] duration:
2.710256503s
FAIL (2025-10-06 23:44:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.710280825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X......XX...X........................................] duration:
2.54266282s
FAIL (2025-10-06 23:44:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.542695061s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X.......X...X..X..X......................................] duration:
2.496964542s
FAIL (2025-10-06 23:44:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.497015018s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
2.574700825s
FAIL (2025-10-06 23:44:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.574734356s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X......XX...X.......................................] duration:
2.727590098s
FAIL (2025-10-06 23:44:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.72768603s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X.......X...X..X..X.....................................] duration:
2.547580159s
FAIL (2025-10-06 23:44:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.547611685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X......XX...X......................................] duration:
2.735263048s
FAIL (2025-10-06 23:43:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.735295963s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....X.......X..X...X.........X..............................] duration:
2.740002436s
FAIL (2025-10-06 23:43:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.740033608s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X.......X...X..X..X....................................] duration:
2.55703306s
FAIL (2025-10-06 23:43:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.557060564s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X......XX...X.....................................] duration:
2.509605687s
FAIL (2025-10-06 23:43:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.509632014s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
2.914335249s
FAIL (2025-10-06 23:43:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.914379648s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X..........X......XX...X....................................] duration:
2.621747679s
FAIL (2025-10-06 23:43:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.621783463s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....X.......X..X...X.........X.............................] duration:
2.807449928s
FAIL (2025-10-06 23:42:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.807531458s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X.......X...X..X..X...................................] duration:
2.787395403s
FAIL (2025-10-06 23:42:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.787431316s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
2.821586583s
FAIL (2025-10-06 23:42:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.821628432s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X..........X......XX...X...................................] duration:
2.778912109s
FAIL (2025-10-06 23:42:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.778956731s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......X.......X..X...X.........X............................] duration:
2.856380017s
FAIL (2025-10-06 23:42:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.856410487s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
2.725214984s
FAIL (2025-10-06 23:42:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.725245983s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X..........X......XX...X..................................] duration:
2.897554387s
FAIL (2025-10-06 23:41:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.897584006s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......X.......X..X...X.........X...........................] duration:
3.576562129s
FAIL (2025-10-06 23:41:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.576589391s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X.......X...X..X..X..................................] duration:
2.663093391s
FAIL (2025-10-06 23:41:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.663123316s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
2.829466982s
FAIL (2025-10-06 23:41:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.829507839s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X..........X......XX...X.................................] duration:
2.843640766s
FAIL (2025-10-06 23:41:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.843667897s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X..........X......XX...X................................] duration:
2.576322447s
FAIL (2025-10-06 23:40:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.576356829s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [X.......X.......X..X...X.........X..........................] duration:
2.961707795s
FAIL (2025-10-06 23:40:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.961747081s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X..........X......XX...X...............................] duration:
2.688482791s
FAIL (2025-10-06 23:40:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.688520704s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X................X..........................................] duration:
2.656336881s
FAIL (2025-10-06 23:40:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.65639206s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X.......X...X..X..X.................................] duration:
2.507926483s
FAIL (2025-10-06 23:40:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.507984541s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X..........X......XX...X..............................] duration:
2.970498268s
FAIL (2025-10-06 23:39:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.97053157s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.X.......X.......X..X...X.........X.........................] duration:
3.063702844s
FAIL (2025-10-06 23:39:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.063730493s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X................X.........................................] duration:
2.811041737s
FAIL (2025-10-06 23:39:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.811064709s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X.......X...X..X..X................................] duration:
2.585576504s
FAIL (2025-10-06 23:39:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.585669448s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X..........X......XX...X.............................] duration:
2.930333479s
FAIL (2025-10-06 23:39:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.930365555s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..X.......X.......X..X...X.........X........................] duration:
3.887118931s
FAIL (2025-10-06 23:38:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.887149091s
core
uplink
us-select-1
us1-placement-20
|