Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X................X.................] duration:
2.588764835s
FAIL (2025-10-06 23:20:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.588784486s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....................X........X..........X......XX...X.......] duration:
2.728516841s
FAIL (2025-10-06 23:20:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.728545747s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........................X.........X.......X...X..X..X......] duration:
2.798098748s
FAIL (2025-10-06 23:20:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.798134945s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........................X.........X.......X...X..X..X.....] duration:
34.339963723s
FAIL (2025-10-06 23:20:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 34.340006711s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X................X................] duration:
27.513139769s
FAIL (2025-10-06 23:20:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.513179896s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....................X........X..........X......XX...X......] duration:
18.386278595s
FAIL (2025-10-06 23:20:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 18.386311809s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........................X.......X.......X..X...X.........X.] duration:
5.496629939s
FAIL (2025-10-06 23:20:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.496672875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..........................X.......X.......X..X...X.........X] duration:
1m0.000703456s
FAIL (2025-10-06 23:20:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000728297s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................X........X..........X......XX...X.....] duration:
2.656255834s
FAIL (2025-10-06 23:18:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.656281545s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X................X...............] duration:
2.545916607s
FAIL (2025-10-06 23:18:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.54594243s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........................X.......X.......X..X...X.........] duration:
3.037297768s
FAIL (2025-10-06 23:18:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.037326512s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........................X.........X.......X...X..X..X....] duration:
2.552934062s
FAIL (2025-10-06 23:18:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.552957148s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......................X........X..........X......XX...X....] duration:
2.659782482s
FAIL (2025-10-06 23:18:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.659814176s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [............................X.......X.......X..X...X........] duration:
2.883824501s
FAIL (2025-10-06 23:17:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.883849248s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.............................X.......X.......X..X...X.......] duration:
3.384026385s
FAIL (2025-10-06 23:17:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.384050038s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X................X..............] duration:
2.816357992s
FAIL (2025-10-06 23:17:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.816707528s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [............................X.........X.......X...X..X..X...] duration:
2.597397056s
FAIL (2025-10-06 23:17:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.597430927s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........................X........X..........X......XX...X...] duration:
2.568816874s
FAIL (2025-10-06 23:16:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.568858355s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..............................X.......X.......X..X...X......] duration:
3.420065318s
FAIL (2025-10-06 23:16:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.420093243s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X................X.............] duration:
2.834191801s
FAIL (2025-10-06 23:16:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.83423395s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............................X.........X.......X...X..X..X..] duration:
2.424291006s
FAIL (2025-10-06 23:16:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.424344664s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...............................X.......X.......X..X...X.....] duration:
3.078264489s
FAIL (2025-10-06 23:16:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.078318711s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........................X........X..........X......XX...X..] duration:
2.613913285s
FAIL (2025-10-06 23:15:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.613944559s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............................X.........X.......X...X..X..X.] duration:
3.000762046s
FAIL (2025-10-06 23:15:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.000783879s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............................X.........X.......X...X..X..X] duration:
1m0.110798012s
FAIL (2025-10-06 23:15:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.110826677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........................X........X..........X......XX...X.] duration:
27.471393969s
FAIL (2025-10-06 23:15:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 27.471425731s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [................................X.......X.......X..X...X....] duration:
9.64671185s
FAIL (2025-10-06 23:15:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.646740588s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X................X............] duration:
3.154041523s
FAIL (2025-10-06 23:15:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.154067765s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........................X........X..........X......XX...X] duration:
30.110430354s
FAIL (2025-10-06 23:15:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.110475882s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X................X...........] duration:
2.540104273s
FAIL (2025-10-06 23:14:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.540134808s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.................................X.......X.......X..X...X...] duration:
3.415207962s
FAIL (2025-10-06 23:14:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.415442621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................X........X..........X......XX...] duration:
2.654342689s
FAIL (2025-10-06 23:14:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:12:04.402444369 (321h41m52.205908951s) [upload=283096]: Starting upload
Oct 6 23:12:04.402517251 (321h41m52.205981833s) [upload=283096]: Done starting upload
Oct 6 23:12:04.402564899 (321h41m52.206029478s) [upload=283096]: Uploading segments...
Oct 6 23:12:04.402894733 (321h41m52.206359325s) [upload=283096]: Got next segment. Inline: false
Oct 6 23:12:04.402984049 (321h41m52.206448651s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:12:04.402991459 (321h41m52.206456059s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:12:04.520647646 (321h41m52.324112243s) [seg_pos={0 0}, upload=283096]: Begin upload segment...
Oct 6 23:12:04.520716661 (321h41m52.324181253s) [seg_pos={0 0}, upload=283096]: Done begin upload segment.
Oct 6 23:12:04.520744626 (321h41m52.324209217s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.520784078 (321h41m52.324248673s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521855811 (321h41m52.325320405s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521945995 (321h41m52.325410587s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.522859575 (321h41m52.326324167s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523045808 (321h41m52.326510405s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523621770 (321h41m52.327086362s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523793965 (321h41m52.327258558s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524362162 (321h41m52.327826753s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524473399 (321h41m52.327938003s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525089374 (321h41m52.328553975s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525434367 (321h41m52.328898962s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525570182 (321h41m52.329034781s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525937890 (321h41m52.329402481s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.526251858 (321h41m52.329716449s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.527313659 (321h41m52.33077831s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:05.152295627 (321h41m52.955760228s) [upload=283096]: Next returned nil segment
Oct 6 23:12:05.152318110 (321h41m52.955782703s) [upload=283096]: Waiting for error group managing segments...
Oct 6 23:12:05.298254976 (321h41m53.101719553s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.329410095 (321h41m53.132874672s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.358275028 (321h41m53.161739613s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.364665984 (321h41m53.168130569s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.371155160 (321h41m53.174619737s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.375113916 (321h41m53.178578493s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.376920109 (321h41m53.180384691s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.377027783 (321h41m53.18049236s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.378958782 (321h41m53.182423373s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.019683053 (321h42m19.823147664s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.551189532 (321h42m20.354654117s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.640293784 (321h42m20.443758361s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:34.098663352 (321h42m21.902127929s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098706066 (321h42m21.902170667s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098832481 (321h42m21.902297058s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098957038 (321h42m21.902421635s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:33254->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 6 23:12:34.098994302 (321h42m21.902458879s) [seg_pos={0 0}, upload=283096]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:12:34.208477021 (321h42m22.011941601s) [upload=283096]: Done uploading segments...
Oct 6 23:12:34.208507873 (321h42m22.011972465s) [upload=283096]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.654395023s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................X.........X.......X...X..X..] duration:
2.590756872s
FAIL (2025-10-06 23:13:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
g_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378338059 (321h41m18.181802671s) [noise=true , address=145.79.227.31:4801 , piece_id=IEWKSCNJJ3W72ENNVNPZZEZ2K4RPVQ5JW3JQAGBFRTATBQQGNS2Q, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378373091 (321h41m18.181837682s) [noise=true , address=77.90.31.21:4893 , piece_id=M4YSXCWWBMZ6FQ5EKWXLMJE57IHWYHV7ST53M6HZYOGFIS72YOKQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378821664 (321h41m18.182286256s) [noise=true , address=145.79.227.46:4636 , piece_id=PXSLP7SHP6YEI7NKTDZKNV5DUM2ZO6FQ5IHQHRGWJYKZRM6GQPWQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379007947 (321h41m18.182472552s) [noise=true , address=77.90.31.19:4869 , piece_id=YJKZBIOHGERIWRYR6ADAXTT7K3JWR2GY6EBOWLKVIR3VYHJ7VKKA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379168362 (321h41m18.182632964s) [noise=true , address=145.79.227.38:4983 , piece_id=Y6U7NONMAF7F3AMV2TSICAUS53US6AHUWBLTJARS5I3CPCY7FN6A, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379517496 (321h41m18.182982087s) [noise=true , address=45.91.116.4:4945 , piece_id=I7SSTDSXNTXWGKJ2OIPPZAWLXBOEKUSUWHXZHATRGMXB7SYNEZKQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379633926 (321h41m18.183098526s) [noise=true , address=145.79.227.33:4878 , piece_id=F6EGIDHPST3BU42SA7M4ALFM47OJT7NSKE4DDZX7BZ4JPBF4D7SA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379861391 (321h41m18.183325982s) [noise=true , address=45.91.116.10:4097 , piece_id=6TIWGAD565NRLJEQHQ6K6DWUX2WNWCJQC36YVWZ6CAGXA5WYU3RQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380220871 (321h41m18.183685466s) [noise=true , address=145.79.227.25:4628 , piece_id=AYPG36RH35OTWEMUTF22PPFC47SJPZOZO2WP2LH2YIWNSNBXADDQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380231772 (321h41m18.18369637s) [noise=true , address=145.79.227.16:4942 , piece_id=VYKQJIIO6DT3JIHUKJ6I4TIKDBX4PPE5ATG3WT36VAUWBLGJ7QFA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380625689 (321h41m18.184090279s) [noise=true , address=77.90.31.15:4728 , piece_id=7XOM5H2AJETWKAO52XIE6EF32SLTYPKTYRNWWS27G3ZN7VDM7JEA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380922997 (321h41m18.184387588s) [noise=true , address=145.79.227.52:4469 , piece_id=4CGM6FSHHICERIRSK4KL5G4YOUSKJZFZIXSQ2GWYDE3SOGSYLBVA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.381201392 (321h41m18.184665983s) [noise=true , address=77.90.31.14:4672 , piece_id=4CJ473CCHKCUUUGLOJ73WJIXQ3HYPK7LPAQ25DYZAOQJLHZJCB2Q, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.381670341 (321h41m18.185134943s) [noise=true , address=64.49.13.138:4484 , piece_id=YFM6A234OBHTRUH6IBM56JQLIRWUESVPNOYNAOX2GX6FIVYSRUJA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.382262240 (321h41m18.185726822s) [noise=true , address=45.91.116.5:4328 , piece_id=37QP3JOD5DKWGQYTK4464PGCFPIGDUBQMLEFCBXA6KQ3HBZHAPBQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.383007119 (321h41m18.186471698s) [noise=true , address=145.79.227.23:4566 , piece_id=ICTF2UJBN5U6NKTVIEI5BXDDBCVSI7FUSRUKH3I5WRBOD5WBWR2A, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.955144639 (321h41m18.758609234s) [upload=283090]: Next returned nil segment
Oct 6 23:11:30.955165503 (321h41m18.758630104s) [upload=283090]: Waiting for error group managing segments...
Oct 6 23:11:31.099220185 (321h41m18.902684778s) [noise=true , address=145.79.227.52:4469 , piece_id=4CGM6FSHHICERIRSK4KL5G4YOUSKJZFZIXSQ2GWYDE3SOGSYLBVA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.100211591 (321h41m18.903676183s) [noise=true , address=145.79.227.24:4613 , piece_id=57KINGN6EJU6NEDFYTLLBYLKPGBSFHLUA5YL3QULCMT5PJTN2MTQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.102716082 (321h41m18.906180667s) [noise=true , address=145.79.227.25:4628 , piece_id=AYPG36RH35OTWEMUTF22PPFC47SJPZOZO2WP2LH2YIWNSNBXADDQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.108484918 (321h41m18.911949502s) [noise=true , address=145.79.227.46:4636 , piece_id=PXSLP7SHP6YEI7NKTDZKNV5DUM2ZO6FQ5IHQHRGWJYKZRM6GQPWQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.115377734 (321h41m18.918842317s) [noise=true , address=145.79.227.31:4801 , piece_id=IEWKSCNJJ3W72ENNVNPZZEZ2K4RPVQ5JW3JQAGBFRTATBQQGNS2Q, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.126638657 (321h41m18.930103235s) [noise=true , address=145.79.227.23:4566 , piece_id=ICTF2UJBN5U6NKTVIEI5BXDDBCVSI7FUSRUKH3I5WRBOD5WBWR2A, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.142628631 (321h41m18.946093228s) [noise=true , address=145.79.227.33:4878 , piece_id=F6EGIDHPST3BU42SA7M4ALFM47OJT7NSKE4DDZX7BZ4JPBF4D7SA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.157575071 (321h41m18.961039663s) [noise=true , address=145.79.227.16:4942 , piece_id=VYKQJIIO6DT3JIHUKJ6I4TIKDBX4PPE5ATG3WT36VAUWBLGJ7QFA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.158618981 (321h41m18.962083558s) [noise=true , address=145.79.227.38:4983 , piece_id=Y6U7NONMAF7F3AMV2TSICAUS53US6AHUWBLTJARS5I3CPCY7FN6A, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.624566176 (321h41m19.428030754s) [noise=true , address=64.49.13.138:4484 , piece_id=YFM6A234OBHTRUH6IBM56JQLIRWUESVPNOYNAOX2GX6FIVYSRUJA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:12:29.953532486 (321h42m17.756997083s) [noise=true , address=45.91.116.5:4328 , piece_id=37QP3JOD5DKWGQYTK4464PGCFPIGDUBQMLEFCBXA6KQ3HBZHAPBQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1271BdWAaWfHQDmGXeg26sz9kXEP9CbokApwJPn9Ng5ZYgFHT1t, address:45.91.116.5:4328): protocol: read tcp 139.59.144.22:34932->45.91.116.5:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.953763031 (321h42m17.757227627s) [noise=true , address=45.91.116.9:4293 , piece_id=6GHQPYLLJFQXSRZUAVNMRC4SUW3FULMC6IFC6UN22GH3ELROZJRA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: read tcp 139.59.144.22:59454->45.91.116.9:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.953978206 (321h42m17.757442787s) [noise=true , address=45.91.116.10:4097 , piece_id=6TIWGAD565NRLJEQHQ6K6DWUX2WNWCJQC36YVWZ6CAGXA5WYU3RQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY, address:45.91.116.10:4097): protocol: read tcp 139.59.144.22:35056->45.91.116.10:4097: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954145516 (321h42m17.757610108s) [noise=true , address=45.91.116.4:4945 , piece_id=I7SSTDSXNTXWGKJ2OIPPZAWLXBOEKUSUWHXZHATRGMXB7SYNEZKQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5, address:45.91.116.4:4945): protocol: read tcp 139.59.144.22:50230->45.91.116.4:4945: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954205013 (321h42m17.757669591s) [noise=true , address=77.90.31.14:4672 , piece_id=4CJ473CCHKCUUUGLOJ73WJIXQ3HYPK7LPAQ25DYZAOQJLHZJCB2Q, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: read tcp 139.59.144.22:35342->77.90.31.14:4672: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954301034 (321h42m17.757765625s) [noise=true , address=77.90.31.15:4728 , piece_id=7XOM5H2AJETWKAO52XIE6EF32SLTYPKTYRNWWS27G3ZN7VDM7JEA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: read tcp 139.59.144.22:55164->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954449762 (321h42m17.757914347s) [noise=true , address=77.90.31.21:4893 , piece_id=M4YSXCWWBMZ6FQ5EKWXLMJE57IHWYHV7ST53M6HZYOGFIS72YOKQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, address:77.90.31.21:4893): protocol: read tcp 139.59.144.22:42284->77.90.31.21:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954471295 (321h42m17.757935876s) [noise=true , address=77.90.31.19:4869 , piece_id=YJKZBIOHGERIWRYR6ADAXTT7K3JWR2GY6EBOWLKVIR3VYHJ7VKKA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: read tcp 139.59.144.22:53234->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954499079 (321h42m17.757963669s) [seg_pos={0 0}, upload=283090]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Oct 6 23:12:30.067681206 (321h42m17.871145805s) [upload=283090]: Done uploading segments...
Oct 6 23:12:30.067716671 (321h42m17.871181269s) [upload=283090]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 2.590797429s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X................X..........] duration:
2.713539269s
FAIL (2025-10-06 23:13:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.713571544s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..................................X.......X.......X..X...X..] duration:
6.350474824s
FAIL (2025-10-06 23:13:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.350501068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...................................X.......X.......X..X...X.] duration:
17.747495301s
FAIL (2025-10-06 23:13:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.747542844s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X................X.........] duration:
3.090894015s
FAIL (2025-10-06 23:13:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.090922521s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............................X........X..........X......XX..] duration:
2.718308695s
FAIL (2025-10-06 23:12:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:12:04.402444369 (321h41m52.205908951s) [upload=283096]: Starting upload
Oct 6 23:12:04.402517251 (321h41m52.205981833s) [upload=283096]: Done starting upload
Oct 6 23:12:04.402564899 (321h41m52.206029478s) [upload=283096]: Uploading segments...
Oct 6 23:12:04.402894733 (321h41m52.206359325s) [upload=283096]: Got next segment. Inline: false
Oct 6 23:12:04.402984049 (321h41m52.206448651s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:12:04.402991459 (321h41m52.206456059s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:12:04.520647646 (321h41m52.324112243s) [seg_pos={0 0}, upload=283096]: Begin upload segment...
Oct 6 23:12:04.520716661 (321h41m52.324181253s) [seg_pos={0 0}, upload=283096]: Done begin upload segment.
Oct 6 23:12:04.520744626 (321h41m52.324209217s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.520784078 (321h41m52.324248673s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521855811 (321h41m52.325320405s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521945995 (321h41m52.325410587s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.522859575 (321h41m52.326324167s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523045808 (321h41m52.326510405s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523621770 (321h41m52.327086362s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523793965 (321h41m52.327258558s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524362162 (321h41m52.327826753s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524473399 (321h41m52.327938003s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525089374 (321h41m52.328553975s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525434367 (321h41m52.328898962s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525570182 (321h41m52.329034781s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525937890 (321h41m52.329402481s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.526251858 (321h41m52.329716449s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.527313659 (321h41m52.33077831s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:05.152295627 (321h41m52.955760228s) [upload=283096]: Next returned nil segment
Oct 6 23:12:05.152318110 (321h41m52.955782703s) [upload=283096]: Waiting for error group managing segments...
Oct 6 23:12:05.298254976 (321h41m53.101719553s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.329410095 (321h41m53.132874672s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.358275028 (321h41m53.161739613s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.364665984 (321h41m53.168130569s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.371155160 (321h41m53.174619737s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.375113916 (321h41m53.178578493s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.376920109 (321h41m53.180384691s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.377027783 (321h41m53.18049236s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.378958782 (321h41m53.182423373s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.019683053 (321h42m19.823147664s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.551189532 (321h42m20.354654117s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.640293784 (321h42m20.443758361s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:34.098663352 (321h42m21.902127929s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098706066 (321h42m21.902170667s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098832481 (321h42m21.902297058s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098957038 (321h42m21.902421635s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:33254->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 6 23:12:34.098994302 (321h42m21.902458879s) [seg_pos={0 0}, upload=283096]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:12:34.208477021 (321h42m22.011941601s) [upload=283096]: Done uploading segments...
Oct 6 23:12:34.208507873 (321h42m22.011972465s) [upload=283096]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.718339143s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....................................X.......X.......X..X...X] duration:
1m0.005661982s
FAIL (2025-10-06 23:12:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.005691042s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............................X........X..........X......XX.] duration:
3.555865202s
FAIL (2025-10-06 23:12:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:12:04.402444369 (321h41m52.205908951s) [upload=283096]: Starting upload
Oct 6 23:12:04.402517251 (321h41m52.205981833s) [upload=283096]: Done starting upload
Oct 6 23:12:04.402564899 (321h41m52.206029478s) [upload=283096]: Uploading segments...
Oct 6 23:12:04.402894733 (321h41m52.206359325s) [upload=283096]: Got next segment. Inline: false
Oct 6 23:12:04.402984049 (321h41m52.206448651s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:12:04.402991459 (321h41m52.206456059s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:12:04.520647646 (321h41m52.324112243s) [seg_pos={0 0}, upload=283096]: Begin upload segment...
Oct 6 23:12:04.520716661 (321h41m52.324181253s) [seg_pos={0 0}, upload=283096]: Done begin upload segment.
Oct 6 23:12:04.520744626 (321h41m52.324209217s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.520784078 (321h41m52.324248673s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521855811 (321h41m52.325320405s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521945995 (321h41m52.325410587s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.522859575 (321h41m52.326324167s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523045808 (321h41m52.326510405s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523621770 (321h41m52.327086362s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523793965 (321h41m52.327258558s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524362162 (321h41m52.327826753s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524473399 (321h41m52.327938003s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525089374 (321h41m52.328553975s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525434367 (321h41m52.328898962s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525570182 (321h41m52.329034781s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525937890 (321h41m52.329402481s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.526251858 (321h41m52.329716449s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.527313659 (321h41m52.33077831s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:05.152295627 (321h41m52.955760228s) [upload=283096]: Next returned nil segment
Oct 6 23:12:05.152318110 (321h41m52.955782703s) [upload=283096]: Waiting for error group managing segments...
Oct 6 23:12:05.298254976 (321h41m53.101719553s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.329410095 (321h41m53.132874672s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.358275028 (321h41m53.161739613s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.364665984 (321h41m53.168130569s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.371155160 (321h41m53.174619737s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.375113916 (321h41m53.178578493s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.376920109 (321h41m53.180384691s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.377027783 (321h41m53.18049236s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.378958782 (321h41m53.182423373s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.019683053 (321h42m19.823147664s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.551189532 (321h42m20.354654117s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.640293784 (321h42m20.443758361s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:34.098663352 (321h42m21.902127929s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098706066 (321h42m21.902170667s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098832481 (321h42m21.902297058s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098957038 (321h42m21.902421635s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:33254->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 6 23:12:34.098994302 (321h42m21.902458879s) [seg_pos={0 0}, upload=283096]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:12:34.208477021 (321h42m22.011941601s) [upload=283096]: Done uploading segments...
Oct 6 23:12:34.208507873 (321h42m22.011972465s) [upload=283096]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.555893869s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............................X........X..........X......XX] duration:
30.111752443s
FAIL (2025-10-06 23:12:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:12:04.402444369 (321h41m52.205908951s) [upload=283096]: Starting upload
Oct 6 23:12:04.402517251 (321h41m52.205981833s) [upload=283096]: Done starting upload
Oct 6 23:12:04.402564899 (321h41m52.206029478s) [upload=283096]: Uploading segments...
Oct 6 23:12:04.402894733 (321h41m52.206359325s) [upload=283096]: Got next segment. Inline: false
Oct 6 23:12:04.402984049 (321h41m52.206448651s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:12:04.402991459 (321h41m52.206456059s) [upload=283096]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:12:04.520647646 (321h41m52.324112243s) [seg_pos={0 0}, upload=283096]: Begin upload segment...
Oct 6 23:12:04.520716661 (321h41m52.324181253s) [seg_pos={0 0}, upload=283096]: Done begin upload segment.
Oct 6 23:12:04.520744626 (321h41m52.324209217s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.520784078 (321h41m52.324248673s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521855811 (321h41m52.325320405s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.521945995 (321h41m52.325410587s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.522859575 (321h41m52.326324167s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523045808 (321h41m52.326510405s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523621770 (321h41m52.327086362s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.523793965 (321h41m52.327258558s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524362162 (321h41m52.327826753s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.524473399 (321h41m52.327938003s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525089374 (321h41m52.328553975s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525434367 (321h41m52.328898962s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525570182 (321h41m52.329034781s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.525937890 (321h41m52.329402481s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.526251858 (321h41m52.329716449s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:04.527313659 (321h41m52.33077831s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Uploading piece...
Oct 6 23:12:05.152295627 (321h41m52.955760228s) [upload=283096]: Next returned nil segment
Oct 6 23:12:05.152318110 (321h41m52.955782703s) [upload=283096]: Waiting for error group managing segments...
Oct 6 23:12:05.298254976 (321h41m53.101719553s) [noise=true , address=145.79.227.14:4837 , piece_id=F7JAHCLRWMJSXVGVC5C5OG6T5RXQKT5CL4CAZXLE4BGFKFXYV5OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.329410095 (321h41m53.132874672s) [noise=true , address=145.79.227.2:4115 , piece_id=BMH7VXPV56KXDSJRL2LDAMCFKD446YGJ5F276I7GXQUVBBMQY3MA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.358275028 (321h41m53.161739613s) [noise=true , address=145.79.227.17:4972 , piece_id=OL47LGCAWTRQGWBFYF5R5LFR7Q33SIL5TC4BO5UYDXIQGTFXKLYA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.364665984 (321h41m53.168130569s) [noise=true , address=145.79.227.14:4840 , piece_id=VPLRJOXNCYVVWN45F7ITZUQUFRSKC5F4OUPFF2YTGVODLIY6XOBQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.371155160 (321h41m53.174619737s) [noise=true , address=145.79.227.13:4801 , piece_id=O622YYTWIW3FACTZA5766EXSL55MT7XSJEMCXHDVJBABZPTVKZKQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.375113916 (321h41m53.178578493s) [noise=true , address=145.79.227.5:4222 , piece_id=SUBBC5B5KTMAUKXW5QFJ6EM5EMUEJXFZVU75KPIAJIA62MSYF3OQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.376920109 (321h41m53.180384691s) [noise=true , address=145.79.227.3:4150 , piece_id=ZU7UBKIGZO2HZHAEDHOJATMSB46BQRBRD27NTWBC6DD3M7LQ7UKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.377027783 (321h41m53.18049236s) [noise=true , address=145.79.227.6:4258 , piece_id=NGYX4JYFMFKZ2LNRWWU4KVQOR332GWCUBJKHEZ6LWGHCQBMKKW7Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:05.378958782 (321h41m53.182423373s) [noise=true , address=145.79.227.11:4730 , piece_id=J3H5UAIYCSDKHYGFJGAJF4NV6XZDGM7YP3VLVSQJRSASLZW26QHA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.019683053 (321h42m19.823147664s) [noise=true , address=45.91.116.7:4010 , piece_id=J7F6KBY67AL4YRE6ONBYODIXS2DGZZKMZJLHJP5GVBS4TKSXCVOA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.551189532 (321h42m20.354654117s) [noise=true , address=45.91.116.13:4154 , piece_id=QBITJR2PCGJDPWI3OPYCJQA3QEEMHTPV74WYNTCFILDKGE2YA5HQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:32.640293784 (321h42m20.443758361s) [noise=true , address=77.90.31.19:4843 , piece_id=TH5MGP5MQ3A2GOC7APIG5C6BOE25EFNMVJHAYNYB7N2TNDPFNUWQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: <nil>
Oct 6 23:12:34.098663352 (321h42m21.902127929s) [noise=true , address=45.91.116.12:4226 , piece_id=27GYE2CY6GSSNSG562LAA3WBF75KM6EKIG7VIYXQ2QO6MU3OIV3Q, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098706066 (321h42m21.902170667s) [noise=true , address=77.90.31.13:4628 , piece_id=HWD7IN2CBPDBJ3ZHO6T4LAWCT7SX55HJ7AUNH6EKRGIWWTGWTHMQ, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098832481 (321h42m21.902297058s) [noise=true , address=77.90.31.10:4520 , piece_id=M26DU2CLMAX5C2DKTI2BNI7EN3L27TSANXPN4FTWIU6YSELGXX7A, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:34.098957038 (321h42m21.902421635s) [noise=true , address=77.90.31.15:4697 , piece_id=5KNADROGCFQ57MAX364QMUEBXWCX5HZAKHCXE43NG7A5AYBRCQKA, seg_pos={0 0}, upload=283096]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: read tcp 139.59.144.22:33254->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Oct 6 23:12:34.098994302 (321h42m21.902458879s) [seg_pos={0 0}, upload=283096]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:12:34.208477021 (321h42m22.011941601s) [upload=283096]: Done uploading segments...
Oct 6 23:12:34.208507873 (321h42m22.011972465s) [upload=283096]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.111791484s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................X.........X.......X...X..X.] duration:
3.92227493s
FAIL (2025-10-06 23:12:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
g_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378338059 (321h41m18.181802671s) [noise=true , address=145.79.227.31:4801 , piece_id=IEWKSCNJJ3W72ENNVNPZZEZ2K4RPVQ5JW3JQAGBFRTATBQQGNS2Q, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378373091 (321h41m18.181837682s) [noise=true , address=77.90.31.21:4893 , piece_id=M4YSXCWWBMZ6FQ5EKWXLMJE57IHWYHV7ST53M6HZYOGFIS72YOKQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378821664 (321h41m18.182286256s) [noise=true , address=145.79.227.46:4636 , piece_id=PXSLP7SHP6YEI7NKTDZKNV5DUM2ZO6FQ5IHQHRGWJYKZRM6GQPWQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379007947 (321h41m18.182472552s) [noise=true , address=77.90.31.19:4869 , piece_id=YJKZBIOHGERIWRYR6ADAXTT7K3JWR2GY6EBOWLKVIR3VYHJ7VKKA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379168362 (321h41m18.182632964s) [noise=true , address=145.79.227.38:4983 , piece_id=Y6U7NONMAF7F3AMV2TSICAUS53US6AHUWBLTJARS5I3CPCY7FN6A, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379517496 (321h41m18.182982087s) [noise=true , address=45.91.116.4:4945 , piece_id=I7SSTDSXNTXWGKJ2OIPPZAWLXBOEKUSUWHXZHATRGMXB7SYNEZKQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379633926 (321h41m18.183098526s) [noise=true , address=145.79.227.33:4878 , piece_id=F6EGIDHPST3BU42SA7M4ALFM47OJT7NSKE4DDZX7BZ4JPBF4D7SA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379861391 (321h41m18.183325982s) [noise=true , address=45.91.116.10:4097 , piece_id=6TIWGAD565NRLJEQHQ6K6DWUX2WNWCJQC36YVWZ6CAGXA5WYU3RQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380220871 (321h41m18.183685466s) [noise=true , address=145.79.227.25:4628 , piece_id=AYPG36RH35OTWEMUTF22PPFC47SJPZOZO2WP2LH2YIWNSNBXADDQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380231772 (321h41m18.18369637s) [noise=true , address=145.79.227.16:4942 , piece_id=VYKQJIIO6DT3JIHUKJ6I4TIKDBX4PPE5ATG3WT36VAUWBLGJ7QFA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380625689 (321h41m18.184090279s) [noise=true , address=77.90.31.15:4728 , piece_id=7XOM5H2AJETWKAO52XIE6EF32SLTYPKTYRNWWS27G3ZN7VDM7JEA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380922997 (321h41m18.184387588s) [noise=true , address=145.79.227.52:4469 , piece_id=4CGM6FSHHICERIRSK4KL5G4YOUSKJZFZIXSQ2GWYDE3SOGSYLBVA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.381201392 (321h41m18.184665983s) [noise=true , address=77.90.31.14:4672 , piece_id=4CJ473CCHKCUUUGLOJ73WJIXQ3HYPK7LPAQ25DYZAOQJLHZJCB2Q, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.381670341 (321h41m18.185134943s) [noise=true , address=64.49.13.138:4484 , piece_id=YFM6A234OBHTRUH6IBM56JQLIRWUESVPNOYNAOX2GX6FIVYSRUJA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.382262240 (321h41m18.185726822s) [noise=true , address=45.91.116.5:4328 , piece_id=37QP3JOD5DKWGQYTK4464PGCFPIGDUBQMLEFCBXA6KQ3HBZHAPBQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.383007119 (321h41m18.186471698s) [noise=true , address=145.79.227.23:4566 , piece_id=ICTF2UJBN5U6NKTVIEI5BXDDBCVSI7FUSRUKH3I5WRBOD5WBWR2A, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.955144639 (321h41m18.758609234s) [upload=283090]: Next returned nil segment
Oct 6 23:11:30.955165503 (321h41m18.758630104s) [upload=283090]: Waiting for error group managing segments...
Oct 6 23:11:31.099220185 (321h41m18.902684778s) [noise=true , address=145.79.227.52:4469 , piece_id=4CGM6FSHHICERIRSK4KL5G4YOUSKJZFZIXSQ2GWYDE3SOGSYLBVA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.100211591 (321h41m18.903676183s) [noise=true , address=145.79.227.24:4613 , piece_id=57KINGN6EJU6NEDFYTLLBYLKPGBSFHLUA5YL3QULCMT5PJTN2MTQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.102716082 (321h41m18.906180667s) [noise=true , address=145.79.227.25:4628 , piece_id=AYPG36RH35OTWEMUTF22PPFC47SJPZOZO2WP2LH2YIWNSNBXADDQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.108484918 (321h41m18.911949502s) [noise=true , address=145.79.227.46:4636 , piece_id=PXSLP7SHP6YEI7NKTDZKNV5DUM2ZO6FQ5IHQHRGWJYKZRM6GQPWQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.115377734 (321h41m18.918842317s) [noise=true , address=145.79.227.31:4801 , piece_id=IEWKSCNJJ3W72ENNVNPZZEZ2K4RPVQ5JW3JQAGBFRTATBQQGNS2Q, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.126638657 (321h41m18.930103235s) [noise=true , address=145.79.227.23:4566 , piece_id=ICTF2UJBN5U6NKTVIEI5BXDDBCVSI7FUSRUKH3I5WRBOD5WBWR2A, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.142628631 (321h41m18.946093228s) [noise=true , address=145.79.227.33:4878 , piece_id=F6EGIDHPST3BU42SA7M4ALFM47OJT7NSKE4DDZX7BZ4JPBF4D7SA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.157575071 (321h41m18.961039663s) [noise=true , address=145.79.227.16:4942 , piece_id=VYKQJIIO6DT3JIHUKJ6I4TIKDBX4PPE5ATG3WT36VAUWBLGJ7QFA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.158618981 (321h41m18.962083558s) [noise=true , address=145.79.227.38:4983 , piece_id=Y6U7NONMAF7F3AMV2TSICAUS53US6AHUWBLTJARS5I3CPCY7FN6A, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.624566176 (321h41m19.428030754s) [noise=true , address=64.49.13.138:4484 , piece_id=YFM6A234OBHTRUH6IBM56JQLIRWUESVPNOYNAOX2GX6FIVYSRUJA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:12:29.953532486 (321h42m17.756997083s) [noise=true , address=45.91.116.5:4328 , piece_id=37QP3JOD5DKWGQYTK4464PGCFPIGDUBQMLEFCBXA6KQ3HBZHAPBQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1271BdWAaWfHQDmGXeg26sz9kXEP9CbokApwJPn9Ng5ZYgFHT1t, address:45.91.116.5:4328): protocol: read tcp 139.59.144.22:34932->45.91.116.5:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.953763031 (321h42m17.757227627s) [noise=true , address=45.91.116.9:4293 , piece_id=6GHQPYLLJFQXSRZUAVNMRC4SUW3FULMC6IFC6UN22GH3ELROZJRA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: read tcp 139.59.144.22:59454->45.91.116.9:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.953978206 (321h42m17.757442787s) [noise=true , address=45.91.116.10:4097 , piece_id=6TIWGAD565NRLJEQHQ6K6DWUX2WNWCJQC36YVWZ6CAGXA5WYU3RQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY, address:45.91.116.10:4097): protocol: read tcp 139.59.144.22:35056->45.91.116.10:4097: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954145516 (321h42m17.757610108s) [noise=true , address=45.91.116.4:4945 , piece_id=I7SSTDSXNTXWGKJ2OIPPZAWLXBOEKUSUWHXZHATRGMXB7SYNEZKQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5, address:45.91.116.4:4945): protocol: read tcp 139.59.144.22:50230->45.91.116.4:4945: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954205013 (321h42m17.757669591s) [noise=true , address=77.90.31.14:4672 , piece_id=4CJ473CCHKCUUUGLOJ73WJIXQ3HYPK7LPAQ25DYZAOQJLHZJCB2Q, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: read tcp 139.59.144.22:35342->77.90.31.14:4672: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954301034 (321h42m17.757765625s) [noise=true , address=77.90.31.15:4728 , piece_id=7XOM5H2AJETWKAO52XIE6EF32SLTYPKTYRNWWS27G3ZN7VDM7JEA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: read tcp 139.59.144.22:55164->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954449762 (321h42m17.757914347s) [noise=true , address=77.90.31.21:4893 , piece_id=M4YSXCWWBMZ6FQ5EKWXLMJE57IHWYHV7ST53M6HZYOGFIS72YOKQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, address:77.90.31.21:4893): protocol: read tcp 139.59.144.22:42284->77.90.31.21:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954471295 (321h42m17.757935876s) [noise=true , address=77.90.31.19:4869 , piece_id=YJKZBIOHGERIWRYR6ADAXTT7K3JWR2GY6EBOWLKVIR3VYHJ7VKKA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: read tcp 139.59.144.22:53234->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954499079 (321h42m17.757963669s) [seg_pos={0 0}, upload=283090]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Oct 6 23:12:30.067681206 (321h42m17.871145805s) [upload=283090]: Done uploading segments...
Oct 6 23:12:30.067716671 (321h42m17.871181269s) [upload=283090]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 3.922298394s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X................X........] duration:
25.834906153s
FAIL (2025-10-06 23:12:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.834941003s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................X.........X.......X...X..X] duration:
1m0.11648907s
FAIL (2025-10-06 23:12:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
g_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378338059 (321h41m18.181802671s) [noise=true , address=145.79.227.31:4801 , piece_id=IEWKSCNJJ3W72ENNVNPZZEZ2K4RPVQ5JW3JQAGBFRTATBQQGNS2Q, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378373091 (321h41m18.181837682s) [noise=true , address=77.90.31.21:4893 , piece_id=M4YSXCWWBMZ6FQ5EKWXLMJE57IHWYHV7ST53M6HZYOGFIS72YOKQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.378821664 (321h41m18.182286256s) [noise=true , address=145.79.227.46:4636 , piece_id=PXSLP7SHP6YEI7NKTDZKNV5DUM2ZO6FQ5IHQHRGWJYKZRM6GQPWQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379007947 (321h41m18.182472552s) [noise=true , address=77.90.31.19:4869 , piece_id=YJKZBIOHGERIWRYR6ADAXTT7K3JWR2GY6EBOWLKVIR3VYHJ7VKKA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379168362 (321h41m18.182632964s) [noise=true , address=145.79.227.38:4983 , piece_id=Y6U7NONMAF7F3AMV2TSICAUS53US6AHUWBLTJARS5I3CPCY7FN6A, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379517496 (321h41m18.182982087s) [noise=true , address=45.91.116.4:4945 , piece_id=I7SSTDSXNTXWGKJ2OIPPZAWLXBOEKUSUWHXZHATRGMXB7SYNEZKQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379633926 (321h41m18.183098526s) [noise=true , address=145.79.227.33:4878 , piece_id=F6EGIDHPST3BU42SA7M4ALFM47OJT7NSKE4DDZX7BZ4JPBF4D7SA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.379861391 (321h41m18.183325982s) [noise=true , address=45.91.116.10:4097 , piece_id=6TIWGAD565NRLJEQHQ6K6DWUX2WNWCJQC36YVWZ6CAGXA5WYU3RQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380220871 (321h41m18.183685466s) [noise=true , address=145.79.227.25:4628 , piece_id=AYPG36RH35OTWEMUTF22PPFC47SJPZOZO2WP2LH2YIWNSNBXADDQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380231772 (321h41m18.18369637s) [noise=true , address=145.79.227.16:4942 , piece_id=VYKQJIIO6DT3JIHUKJ6I4TIKDBX4PPE5ATG3WT36VAUWBLGJ7QFA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380625689 (321h41m18.184090279s) [noise=true , address=77.90.31.15:4728 , piece_id=7XOM5H2AJETWKAO52XIE6EF32SLTYPKTYRNWWS27G3ZN7VDM7JEA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.380922997 (321h41m18.184387588s) [noise=true , address=145.79.227.52:4469 , piece_id=4CGM6FSHHICERIRSK4KL5G4YOUSKJZFZIXSQ2GWYDE3SOGSYLBVA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.381201392 (321h41m18.184665983s) [noise=true , address=77.90.31.14:4672 , piece_id=4CJ473CCHKCUUUGLOJ73WJIXQ3HYPK7LPAQ25DYZAOQJLHZJCB2Q, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.381670341 (321h41m18.185134943s) [noise=true , address=64.49.13.138:4484 , piece_id=YFM6A234OBHTRUH6IBM56JQLIRWUESVPNOYNAOX2GX6FIVYSRUJA, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.382262240 (321h41m18.185726822s) [noise=true , address=45.91.116.5:4328 , piece_id=37QP3JOD5DKWGQYTK4464PGCFPIGDUBQMLEFCBXA6KQ3HBZHAPBQ, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.383007119 (321h41m18.186471698s) [noise=true , address=145.79.227.23:4566 , piece_id=ICTF2UJBN5U6NKTVIEI5BXDDBCVSI7FUSRUKH3I5WRBOD5WBWR2A, seg_pos={0 0}, upload=283090]: Uploading piece...
Oct 6 23:11:30.955144639 (321h41m18.758609234s) [upload=283090]: Next returned nil segment
Oct 6 23:11:30.955165503 (321h41m18.758630104s) [upload=283090]: Waiting for error group managing segments...
Oct 6 23:11:31.099220185 (321h41m18.902684778s) [noise=true , address=145.79.227.52:4469 , piece_id=4CGM6FSHHICERIRSK4KL5G4YOUSKJZFZIXSQ2GWYDE3SOGSYLBVA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.100211591 (321h41m18.903676183s) [noise=true , address=145.79.227.24:4613 , piece_id=57KINGN6EJU6NEDFYTLLBYLKPGBSFHLUA5YL3QULCMT5PJTN2MTQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.102716082 (321h41m18.906180667s) [noise=true , address=145.79.227.25:4628 , piece_id=AYPG36RH35OTWEMUTF22PPFC47SJPZOZO2WP2LH2YIWNSNBXADDQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.108484918 (321h41m18.911949502s) [noise=true , address=145.79.227.46:4636 , piece_id=PXSLP7SHP6YEI7NKTDZKNV5DUM2ZO6FQ5IHQHRGWJYKZRM6GQPWQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.115377734 (321h41m18.918842317s) [noise=true , address=145.79.227.31:4801 , piece_id=IEWKSCNJJ3W72ENNVNPZZEZ2K4RPVQ5JW3JQAGBFRTATBQQGNS2Q, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.126638657 (321h41m18.930103235s) [noise=true , address=145.79.227.23:4566 , piece_id=ICTF2UJBN5U6NKTVIEI5BXDDBCVSI7FUSRUKH3I5WRBOD5WBWR2A, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.142628631 (321h41m18.946093228s) [noise=true , address=145.79.227.33:4878 , piece_id=F6EGIDHPST3BU42SA7M4ALFM47OJT7NSKE4DDZX7BZ4JPBF4D7SA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.157575071 (321h41m18.961039663s) [noise=true , address=145.79.227.16:4942 , piece_id=VYKQJIIO6DT3JIHUKJ6I4TIKDBX4PPE5ATG3WT36VAUWBLGJ7QFA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.158618981 (321h41m18.962083558s) [noise=true , address=145.79.227.38:4983 , piece_id=Y6U7NONMAF7F3AMV2TSICAUS53US6AHUWBLTJARS5I3CPCY7FN6A, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:11:31.624566176 (321h41m19.428030754s) [noise=true , address=64.49.13.138:4484 , piece_id=YFM6A234OBHTRUH6IBM56JQLIRWUESVPNOYNAOX2GX6FIVYSRUJA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: <nil>
Oct 6 23:12:29.953532486 (321h42m17.756997083s) [noise=true , address=45.91.116.5:4328 , piece_id=37QP3JOD5DKWGQYTK4464PGCFPIGDUBQMLEFCBXA6KQ3HBZHAPBQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1271BdWAaWfHQDmGXeg26sz9kXEP9CbokApwJPn9Ng5ZYgFHT1t, address:45.91.116.5:4328): protocol: read tcp 139.59.144.22:34932->45.91.116.5:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.953763031 (321h42m17.757227627s) [noise=true , address=45.91.116.9:4293 , piece_id=6GHQPYLLJFQXSRZUAVNMRC4SUW3FULMC6IFC6UN22GH3ELROZJRA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: read tcp 139.59.144.22:59454->45.91.116.9:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.953978206 (321h42m17.757442787s) [noise=true , address=45.91.116.10:4097 , piece_id=6TIWGAD565NRLJEQHQ6K6DWUX2WNWCJQC36YVWZ6CAGXA5WYU3RQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY, address:45.91.116.10:4097): protocol: read tcp 139.59.144.22:35056->45.91.116.10:4097: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954145516 (321h42m17.757610108s) [noise=true , address=45.91.116.4:4945 , piece_id=I7SSTDSXNTXWGKJ2OIPPZAWLXBOEKUSUWHXZHATRGMXB7SYNEZKQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5, address:45.91.116.4:4945): protocol: read tcp 139.59.144.22:50230->45.91.116.4:4945: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954205013 (321h42m17.757669591s) [noise=true , address=77.90.31.14:4672 , piece_id=4CJ473CCHKCUUUGLOJ73WJIXQ3HYPK7LPAQ25DYZAOQJLHZJCB2Q, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1jvNGA554FNdsMYESMmDnnsLmpteXcPbuiEPAZgMnngTgBNerx, address:77.90.31.14:4672): protocol: read tcp 139.59.144.22:35342->77.90.31.14:4672: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954301034 (321h42m17.757765625s) [noise=true , address=77.90.31.15:4728 , piece_id=7XOM5H2AJETWKAO52XIE6EF32SLTYPKTYRNWWS27G3ZN7VDM7JEA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: read tcp 139.59.144.22:55164->77.90.31.15:4728: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954449762 (321h42m17.757914347s) [noise=true , address=77.90.31.21:4893 , piece_id=M4YSXCWWBMZ6FQ5EKWXLMJE57IHWYHV7ST53M6HZYOGFIS72YOKQ, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, address:77.90.31.21:4893): protocol: read tcp 139.59.144.22:42284->77.90.31.21:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954471295 (321h42m17.757935876s) [noise=true , address=77.90.31.19:4869 , piece_id=YJKZBIOHGERIWRYR6ADAXTT7K3JWR2GY6EBOWLKVIR3VYHJ7VKKA, seg_pos={0 0}, upload=283090]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: read tcp 139.59.144.22:53234->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:29.954499079 (321h42m17.757963669s) [seg_pos={0 0}, upload=283090]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Oct 6 23:12:30.067681206 (321h42m17.871145805s) [upload=283090]: Done uploading segments...
Oct 6 23:12:30.067716671 (321h42m17.871181269s) [upload=283090]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.116518022s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................X........X..........X......X] duration:
30.112182036s
FAIL (2025-10-06 23:12:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 6 23:11:34.131640759 (321h41m21.935105352s) [upload=283091]: Starting upload
Oct 6 23:11:34.131750777 (321h41m21.935215376s) [upload=283091]: Done starting upload
Oct 6 23:11:34.131968925 (321h41m21.935433509s) [upload=283091]: Uploading segments...
Oct 6 23:11:34.132216060 (321h41m21.93568065s) [upload=283091]: Got next segment. Inline: false
Oct 6 23:11:34.132256974 (321h41m21.935721555s) [upload=283091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:11:34.132262509 (321h41m21.935727094s) [upload=283091]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:11:34.245511631 (321h41m22.04897628s) [seg_pos={0 0}, upload=283091]: Begin upload segment...
Oct 6 23:11:34.245598316 (321h41m22.049062898s) [seg_pos={0 0}, upload=283091]: Done begin upload segment.
Oct 6 23:11:34.245653559 (321h41m22.04911814s) [noise=true , address=145.79.227.11:4732 , piece_id=U3NCBJK6DUE7UEXH3ODHHMX2L6PIUGN4UONWV2TUVVH556SFHEMQ, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.249978944 (321h41m22.053443565s) [noise=true , address=145.79.227.2:4114 , piece_id=ERZ3GZEWBK6HJMTXGQAYVAO42NCODACEDA56FMDPMMPQ46R25BKA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.250800552 (321h41m22.054265152s) [noise=true , address=145.79.227.17:4970 , piece_id=YCM7QEJWCNFITKY2MOJDYXIO4Q6NNJQA5MM6ZE2QBTNPYH64YXVA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.251367954 (321h41m22.054832534s) [noise=true , address=145.79.227.18:4007 , piece_id=LP5VTVPOJ6JX5ZNPUAYQJEXXBVOAEEOJEVDHMCEIJTSDUV7L3YIA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.252615724 (321h41m22.056080302s) [noise=true , address=145.79.227.7:4295 , piece_id=IRNBOGH7YEMKRV2MJNJMAGKANBPHOSHL3XSP3S2SKVY2IL6RY4HA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.252821672 (321h41m22.056286269s) [noise=true , address=145.79.227.13:4801 , piece_id=6QB4HCTML3P3OHTHF4YQD6SXUWSAFM6BOCMY64XYKK7QOUP3F3QA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.253695360 (321h41m22.057159937s) [noise=true , address=77.90.31.19:4843 , piece_id=TNXJTCATPRJS4F4EVJWJGV4DTW3RO3SFWZ3OVWQP5PLKPEQH6GDA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.254320047 (321h41m22.057784641s) [noise=true , address=45.91.116.10:4082 , piece_id=ZCYZV25AHCYER6SSBCJPIWQUC3G4JE3ATN345KIKWGYF3YOSTF7Q, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.254928660 (321h41m22.058393237s) [noise=true , address=145.79.227.4:4187 , piece_id=FFDXHQYS7FJTWICZKWSF454G7KEUMWBMT6FSGH6O5AQCMAQUZ73A, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.255459043 (321h41m22.058923639s) [noise=true , address=77.90.31.11:4555 , piece_id=LLRICI42GI6OFLCRMFS2JL7Q2WOAOZHLKOD5P4Q5T2USLUT27QFA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.256203147 (321h41m22.059667741s) [noise=true , address=77.90.31.14:4663 , piece_id=54BS7HI3VCNUJ6Y7JZSE4ZEM53NOS7QA7HXZ3OPKEEHUM5C5X77A, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.256789809 (321h41m22.060254407s) [noise=true , address=145.79.227.2:4115 , piece_id=2L6DJC6SZVVYZSXBHHIWWH6VRR2DMKTDDKR5IVSF6RMVE7BESCHA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.257519568 (321h41m22.060984146s) [noise=true , address=145.79.227.14:4840 , piece_id=MCC3XYJA3K44MQGMUABEM4UNB2LNVFDHOXA37T3AG4PYA6QHT4ZQ, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.258092618 (321h41m22.061557213s) [noise=true , address=145.79.227.5:4221 , piece_id=HV3AJVNEYKQXWMZWITICHWVSJNEI5Z7FQC7COXDAF6VYU5BHJUOQ, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.258853579 (321h41m22.062318156s) [noise=true , address=145.79.227.6:4259 , piece_id=PGYRAG5ZKKA4PUVEHYO34QMJFNNIPJNPFWTH7YETTISB2QPGUKYQ, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.259494142 (321h41m22.062958736s) [noise=true , address=45.91.116.14:4191 , piece_id=DVRUEAB5TTD6CA6ZNSUQKGSQ2GU5LHYPCV54PKYKHRCPWTUJNDZA, seg_pos={0 0}, upload=283091]: Uploading piece...
Oct 6 23:11:34.901509947 (321h41m22.704974528s) [upload=283091]: Next returned nil segment
Oct 6 23:11:34.901527441 (321h41m22.704992033s) [upload=283091]: Waiting for error group managing segments...
Oct 6 23:11:35.028619052 (321h41m22.832083639s) [noise=true , address=145.79.227.14:4840 , piece_id=MCC3XYJA3K44MQGMUABEM4UNB2LNVFDHOXA37T3AG4PYA6QHT4ZQ, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.029213842 (321h41m22.832678429s) [noise=true , address=145.79.227.11:4732 , piece_id=U3NCBJK6DUE7UEXH3ODHHMX2L6PIUGN4UONWV2TUVVH556SFHEMQ, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.030315088 (321h41m22.833779673s) [noise=true , address=145.79.227.4:4187 , piece_id=FFDXHQYS7FJTWICZKWSF454G7KEUMWBMT6FSGH6O5AQCMAQUZ73A, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.031157633 (321h41m22.834622217s) [noise=true , address=145.79.227.5:4221 , piece_id=HV3AJVNEYKQXWMZWITICHWVSJNEI5Z7FQC7COXDAF6VYU5BHJUOQ, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.032649739 (321h41m22.83611432s) [noise=true , address=145.79.227.6:4259 , piece_id=PGYRAG5ZKKA4PUVEHYO34QMJFNNIPJNPFWTH7YETTISB2QPGUKYQ, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.033876153 (321h41m22.83734073s) [noise=true , address=145.79.227.17:4970 , piece_id=YCM7QEJWCNFITKY2MOJDYXIO4Q6NNJQA5MM6ZE2QBTNPYH64YXVA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.034014824 (321h41m22.83747941s) [noise=true , address=145.79.227.13:4801 , piece_id=6QB4HCTML3P3OHTHF4YQD6SXUWSAFM6BOCMY64XYKK7QOUP3F3QA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.035595343 (321h41m22.83905992s) [noise=true , address=145.79.227.18:4007 , piece_id=LP5VTVPOJ6JX5ZNPUAYQJEXXBVOAEEOJEVDHMCEIJTSDUV7L3YIA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.035954331 (321h41m22.839418914s) [noise=true , address=145.79.227.7:4295 , piece_id=IRNBOGH7YEMKRV2MJNJMAGKANBPHOSHL3XSP3S2SKVY2IL6RY4HA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.045700607 (321h41m22.849165191s) [noise=true , address=145.79.227.2:4115 , piece_id=2L6DJC6SZVVYZSXBHHIWWH6VRR2DMKTDDKR5IVSF6RMVE7BESCHA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:11:35.050701408 (321h41m22.854165991s) [noise=true , address=145.79.227.2:4114 , piece_id=ERZ3GZEWBK6HJMTXGQAYVAO42NCODACEDA56FMDPMMPQ46R25BKA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: <nil>
Oct 6 23:12:03.837428071 (321h41m51.640892672s) [noise=true , address=77.90.31.19:4843 , piece_id=TNXJTCATPRJS4F4EVJWJGV4DTW3RO3SFWZ3OVWQP5PLKPEQH6GDA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: rpc: dial tcp 77.90.31.19:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:03.837662322 (321h41m51.641126919s) [noise=true , address=77.90.31.11:4555 , piece_id=LLRICI42GI6OFLCRMFS2JL7Q2WOAOZHLKOD5P4Q5T2USLUT27QFA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 139.59.144.22:58226->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:03.837729446 (321h41m51.641194025s) [noise=true , address=77.90.31.14:4663 , piece_id=54BS7HI3VCNUJ6Y7JZSE4ZEM53NOS7QA7HXZ3OPKEEHUM5C5X77A, seg_pos={0 0}, upload=283091]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 139.59.144.22:38002->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:03.837784344 (321h41m51.641248923s) [noise=true , address=45.91.116.10:4082 , piece_id=ZCYZV25AHCYER6SSBCJPIWQUC3G4JE3ATN345KIKWGYF3YOSTF7Q, seg_pos={0 0}, upload=283091]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 139.59.144.22:60344->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:12:03.837950519 (321h41m51.641415112s) [noise=true , address=45.91.116.14:4191 , piece_id=DVRUEAB5TTD6CA6ZNSUQKGSQ2GU5LHYPCV54PKYKHRCPWTUJNDZA, seg_pos={0 0}, upload=283091]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:42162->45.91.116.14:4191: use of closed network connection; context deadline exceeded
Oct 6 23:12:03.838016018 (321h41m51.641480618s) [seg_pos={0 0}, upload=283091]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 6 23:12:03.947135234 (321h41m51.750599829s) [upload=283091]: Done uploading segments...
Oct 6 23:12:03.947185730 (321h41m51.750650333s) [upload=283091]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.112213048s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X................X.......] duration:
2.824469053s
FAIL (2025-10-06 23:11:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.824495046s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X........X..........X......] duration:
2.664662812s
FAIL (2025-10-06 23:11:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:05:54.374177603 (321h35m42.177642185s) [upload=283012]: Starting upload
Oct 6 23:05:54.374258608 (321h35m42.177723263s) [upload=283012]: Done starting upload
Oct 6 23:05:54.374363362 (321h35m42.177827943s) [upload=283012]: Uploading segments...
Oct 6 23:05:54.374565084 (321h35m42.17802967s) [upload=283012]: Got next segment. Inline: false
Oct 6 23:05:54.374619416 (321h35m42.178083998s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:05:54.374625198 (321h35m42.17808979s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:05:54.494188018 (321h35m42.297652597s) [seg_pos={0 0}, upload=283012]: Begin upload segment...
Oct 6 23:05:54.494255877 (321h35m42.297720454s) [seg_pos={0 0}, upload=283012]: Done begin upload segment.
Oct 6 23:05:54.494282957 (321h35m42.297747547s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.494478695 (321h35m42.297943272s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.495695762 (321h35m42.299160339s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496054163 (321h35m42.29951874s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496262227 (321h35m42.299726804s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496659099 (321h35m42.300123684s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496874139 (321h35m42.300338718s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497269607 (321h35m42.300734191s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497383761 (321h35m42.30084834s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497743006 (321h35m42.301207591s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.503300578 (321h35m42.306765164s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.505042243 (321h35m42.308506821s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.506394686 (321h35m42.309859265s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.507734094 (321h35m42.311198672s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.509246867 (321h35m42.312711444s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.510277459 (321h35m42.313742037s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:55.132006582 (321h35m42.935471168s) [upload=283012]: Next returned nil segment
Oct 6 23:05:55.132049300 (321h35m42.935513892s) [upload=283012]: Waiting for error group managing segments...
Oct 6 23:05:55.276997326 (321h35m43.080461904s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277505636 (321h35m43.080970214s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277891232 (321h35m43.081355816s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.278214929 (321h35m43.08167952s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.282176566 (321h35m43.085641143s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283823631 (321h35m43.087288212s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283856266 (321h35m43.087320857s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284344186 (321h35m43.087808767s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284597364 (321h35m43.088061956s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.289948030 (321h35m43.09341261s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.290567653 (321h35m43.094032245s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.294140755 (321h35m43.097605346s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:06:24.065676552 (321h36m11.869141133s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:41710->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065700549 (321h36m11.869165133s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:37320->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065859567 (321h36m11.869324164s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 139.59.144.22:33926->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066062619 (321h36m11.869527291s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066099109 (321h36m11.869563688s) [seg_pos={0 0}, upload=283012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:06:24.180602946 (321h36m11.984067532s) [upload=283012]: Done uploading segments...
Oct 6 23:06:24.180634103 (321h36m11.984098694s) [upload=283012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.664696561s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....................................X.......X.......X..X...] duration:
3.581191128s
FAIL (2025-10-06 23:11:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.58122218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................X.........X.......X...X..] duration:
2.725139879s
FAIL (2025-10-06 23:10:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
em_SegmentBegin
Oct 6 23:08:40.221054472 (321h38m28.024519074s) [seg_pos={0 0}, upload=283046]: Begin upload segment...
Oct 6 23:08:40.221140743 (321h38m28.024605343s) [seg_pos={0 0}, upload=283046]: Done begin upload segment.
Oct 6 23:08:40.221179992 (321h38m28.024644596s) [noise=true , address=45.91.116.9:4279 , piece_id=NYYJ2SEHL7JCIXLQRADKI6M7VAFA4RR562R4UTOJQKVYXNPFIQQQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.221561686 (321h38m28.025026345s) [noise=true , address=45.91.116.7:4042 , piece_id=XJGKGOTSM3M2RAQJQIDUNZLITKNTXQAIO6FLEZHMT2SS6UOBCGJA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.221997629 (321h38m28.025462231s) [noise=true , address=77.90.31.14:4666 , piece_id=PH2E7YWJED4BKANBB47J6FH3G2FJ4JMUNRDNYPGCMVH3N3PVBODA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222178974 (321h38m28.025643573s) [noise=true , address=77.90.31.14:4670 , piece_id=YL752M75GI7SX243KE6RPJOP5NNGVYGRIII6RDRCGELPFWKXTWFQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222525559 (321h38m28.025990158s) [noise=true , address=45.91.116.12:4237 , piece_id=NQGKFSPRP2SFX4AGCQP65PTT3KE3EURM3N2WR7Y5WUBKXY73YMOQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222640484 (321h38m28.026105083s) [noise=true , address=77.90.31.12:4597 , piece_id=FZVEBDTBYTJJNL5UPWZNU66NFCB7JCURX6B5PT63ZJTMZ5IHHGEQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.222988014 (321h38m28.026452613s) [noise=true , address=77.90.31.11:4567 , piece_id=6YVTKLJDUCAIJKNJVDIOVRY3A23HQ32ZBU5AIFXY2HQZSPQBH7AA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223000499 (321h38m28.02646509s) [noise=true , address=145.79.227.31:4826 , piece_id=OIQNDG2O7TWVOBD3S6RNZZWOUH3BSAMYMEXUQQKIJN74ERO5DYBA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223299771 (321h38m28.026764368s) [noise=true , address=145.79.227.32:4836 , piece_id=4HVPB2CGBBY53R5RCMSR4L46XDLYFQHWYNV3FLPTRKAGGAKVHJPQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223453729 (321h38m28.026918327s) [noise=true , address=145.79.227.44:4211 , piece_id=GMMDVRSGSOOP757M3YL5R3AMTSGB4VXCPJFB3R4UP6BFEBESKIDQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223602162 (321h38m28.02706674s) [noise=true , address=77.90.31.15:4729 , piece_id=MQQYYZZUMR4C5VFVYICXT4SI36TBN4FVJTCPEQZO5ANAUVM3IC5Q, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223897567 (321h38m28.027362144s) [noise=true , address=145.79.227.36:4529 , piece_id=T57W2ET5YCHYUAZDPT6J5HTQB23IWQHQ52UE7KJ5JYALKJ6QIKNQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.223939713 (321h38m28.027404321s) [noise=true , address=145.79.227.19:4098 , piece_id=G46UTN55KJYTAISVC3EPQKII7OROGDXIZKISMTLA2BPRL3K3HX7A, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.224196097 (321h38m28.027660688s) [noise=true , address=145.79.227.43:4182 , piece_id=CAMXXP65XWDQMMCMMAR7RA6V4T2APXPCUC6A6YOCNJSVKVAI4M3Q, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.224688975 (321h38m28.028153577s) [noise=true , address=145.79.227.30:4765 , piece_id=JSQFUW2PB433TYWTOUH55URDJ6WRZXAYCK2GXZ2Q4DHQNKUCNQAQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.225193126 (321h38m28.028657722s) [noise=true , address=77.90.31.21:4910 , piece_id=SHFVO6QD7UXHG6AXXBIUSEIS2AQXKCVDR5VUOGJHYFY3RASYLOJA, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.225676830 (321h38m28.029141431s) [noise=true , address=145.79.227.38:4996 , piece_id=BD5WCK2SAFOA2QFJAXQR47YQFZIXCBOKE4QTP5GF6K5FTUVBDJ5A, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.227266946 (321h38m28.030731538s) [noise=true , address=145.79.227.31:4830 , piece_id=GKRSJ3BK3IKSOWLT4J4OKZYDSW3Q4EC5W2ZX2EZESPJDGZUA57PQ, seg_pos={0 0}, upload=283046]: Uploading piece...
Oct 6 23:08:40.799357531 (321h38m28.602822123s) [upload=283046]: Next returned nil segment
Oct 6 23:08:40.799382339 (321h38m28.602846931s) [upload=283046]: Waiting for error group managing segments...
Oct 6 23:08:40.948963327 (321h38m28.752427904s) [noise=true , address=145.79.227.19:4098 , piece_id=G46UTN55KJYTAISVC3EPQKII7OROGDXIZKISMTLA2BPRL3K3HX7A, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.959420973 (321h38m28.762885561s) [noise=true , address=145.79.227.30:4765 , piece_id=JSQFUW2PB433TYWTOUH55URDJ6WRZXAYCK2GXZ2Q4DHQNKUCNQAQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.961318418 (321h38m28.764782996s) [noise=true , address=145.79.227.44:4211 , piece_id=GMMDVRSGSOOP757M3YL5R3AMTSGB4VXCPJFB3R4UP6BFEBESKIDQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.964394500 (321h38m28.767859083s) [noise=true , address=145.79.227.38:4996 , piece_id=BD5WCK2SAFOA2QFJAXQR47YQFZIXCBOKE4QTP5GF6K5FTUVBDJ5A, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.968318821 (321h38m28.7717834s) [noise=true , address=145.79.227.36:4529 , piece_id=T57W2ET5YCHYUAZDPT6J5HTQB23IWQHQ52UE7KJ5JYALKJ6QIKNQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.969548608 (321h38m28.773013193s) [noise=true , address=145.79.227.43:4182 , piece_id=CAMXXP65XWDQMMCMMAR7RA6V4T2APXPCUC6A6YOCNJSVKVAI4M3Q, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.979944778 (321h38m28.783409359s) [noise=true , address=145.79.227.32:4836 , piece_id=4HVPB2CGBBY53R5RCMSR4L46XDLYFQHWYNV3FLPTRKAGGAKVHJPQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:40.981446575 (321h38m28.784911157s) [noise=true , address=145.79.227.31:4826 , piece_id=OIQNDG2O7TWVOBD3S6RNZZWOUH3BSAMYMEXUQQKIJN74ERO5DYBA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:08:41.447405249 (321h38m29.250869827s) [noise=true , address=145.79.227.31:4830 , piece_id=GKRSJ3BK3IKSOWLT4J4OKZYDSW3Q4EC5W2ZX2EZESPJDGZUA57PQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:30.379124581 (321h39m18.182589158s) [noise=true , address=45.91.116.7:4042 , piece_id=XJGKGOTSM3M2RAQJQIDUNZLITKNTXQAIO6FLEZHMT2SS6UOBCGJA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:30.562164117 (321h39m18.36562871s) [noise=true , address=77.90.31.12:4597 , piece_id=FZVEBDTBYTJJNL5UPWZNU66NFCB7JCURX6B5PT63ZJTMZ5IHHGEQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:37.709003735 (321h39m25.512468338s) [noise=true , address=77.90.31.14:4666 , piece_id=PH2E7YWJED4BKANBB47J6FH3G2FJ4JMUNRDNYPGCMVH3N3PVBODA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: <nil>
Oct 6 23:09:39.790162022 (321h39m27.593626605s) [noise=true , address=77.90.31.14:4670 , piece_id=YL752M75GI7SX243KE6RPJOP5NNGVYGRIII6RDRCGELPFWKXTWFQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12LQ2JYC2GB67iqFPW69DyASSKH9yN558jGYBFd2cRN8abegiTE, address:77.90.31.14:4670): protocol: read tcp 139.59.144.22:53666->77.90.31.14:4670: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790251732 (321h39m27.593716324s) [noise=true , address=45.91.116.12:4237 , piece_id=NQGKFSPRP2SFX4AGCQP65PTT3KE3EURM3N2WR7Y5WUBKXY73YMOQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790396915 (321h39m27.593861495s) [noise=true , address=77.90.31.21:4910 , piece_id=SHFVO6QD7UXHG6AXXBIUSEIS2AQXKCVDR5VUOGJHYFY3RASYLOJA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: read tcp 139.59.144.22:36098->77.90.31.21:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790433348 (321h39m27.593897939s) [noise=true , address=77.90.31.15:4729 , piece_id=MQQYYZZUMR4C5VFVYICXT4SI36TBN4FVJTCPEQZO5ANAUVM3IC5Q, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: rpc: dial tcp 77.90.31.15:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790575035 (321h39m27.59403963s) [noise=true , address=45.91.116.9:4279 , piece_id=NYYJ2SEHL7JCIXLQRADKI6M7VAFA4RR562R4UTOJQKVYXNPFIQQQ, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: read tcp 139.59.144.22:47230->45.91.116.9:4279: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790745323 (321h39m27.594209918s) [noise=true , address=77.90.31.11:4567 , piece_id=6YVTKLJDUCAIJKNJVDIOVRY3A23HQ32ZBU5AIFXY2HQZSPQBH7AA, seg_pos={0 0}, upload=283046]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: read tcp 139.59.144.22:48886->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:09:39.790801310 (321h39m27.594265905s) [seg_pos={0 0}, upload=283046]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:09:39.903583697 (321h39m27.707048276s) [upload=283046]: Done uploading segments...
Oct 6 23:09:39.903609834 (321h39m27.707074412s) [upload=283046]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 2.725175117s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X........X..........X.....] duration:
2.500835435s
FAIL (2025-10-06 23:10:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 6 23:05:54.374177603 (321h35m42.177642185s) [upload=283012]: Starting upload
Oct 6 23:05:54.374258608 (321h35m42.177723263s) [upload=283012]: Done starting upload
Oct 6 23:05:54.374363362 (321h35m42.177827943s) [upload=283012]: Uploading segments...
Oct 6 23:05:54.374565084 (321h35m42.17802967s) [upload=283012]: Got next segment. Inline: false
Oct 6 23:05:54.374619416 (321h35m42.178083998s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:05:54.374625198 (321h35m42.17808979s) [upload=283012]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:05:54.494188018 (321h35m42.297652597s) [seg_pos={0 0}, upload=283012]: Begin upload segment...
Oct 6 23:05:54.494255877 (321h35m42.297720454s) [seg_pos={0 0}, upload=283012]: Done begin upload segment.
Oct 6 23:05:54.494282957 (321h35m42.297747547s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.494478695 (321h35m42.297943272s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.495695762 (321h35m42.299160339s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496054163 (321h35m42.29951874s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496262227 (321h35m42.299726804s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496659099 (321h35m42.300123684s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.496874139 (321h35m42.300338718s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497269607 (321h35m42.300734191s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497383761 (321h35m42.30084834s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.497743006 (321h35m42.301207591s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.503300578 (321h35m42.306765164s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.505042243 (321h35m42.308506821s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.506394686 (321h35m42.309859265s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.507734094 (321h35m42.311198672s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.509246867 (321h35m42.312711444s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:54.510277459 (321h35m42.313742037s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Uploading piece...
Oct 6 23:05:55.132006582 (321h35m42.935471168s) [upload=283012]: Next returned nil segment
Oct 6 23:05:55.132049300 (321h35m42.935513892s) [upload=283012]: Waiting for error group managing segments...
Oct 6 23:05:55.276997326 (321h35m43.080461904s) [noise=true , address=145.79.227.14:4839 , piece_id=UJB6UBBKVPVM532RDFFAX5NYDODVR4XMYOVEP4Q444FXB2SL5HSQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277505636 (321h35m43.080970214s) [noise=true , address=145.79.227.16:4909 , piece_id=SVHEB4ISXKWYEF4WH4OMVUC6XQ5Z5D6PMKTUCM3744FXSBPVGHWA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.277891232 (321h35m43.081355816s) [noise=true , address=145.79.227.15:4876 , piece_id=QZCU6F4HE2LISAH6LTJBSC62PYIPF3KUTXEX2DCWVH2VRKF633QA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.278214929 (321h35m43.08167952s) [noise=true , address=145.79.227.11:4732 , piece_id=LTCKW4N637APLT6QDVYQJXYWLLUTBNPIKD3JNRDZ7XKELVFCLEDA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.282176566 (321h35m43.085641143s) [noise=true , address=145.79.227.13:4802 , piece_id=7JY4M67VIIWR36O7KHBQPYRZ5ZVMZTQHI7SME3Q2XV3ZPLHEXP6Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283823631 (321h35m43.087288212s) [noise=true , address=145.79.227.3:4151 , piece_id=M44VBXFT3ILYG4OVOP32WJ3DZDBQBADGWR7S5S2JL3JQLLG4CLTQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.283856266 (321h35m43.087320857s) [noise=true , address=145.79.227.2:4113 , piece_id=7GK2OZKO23BZQ5KOR32SYU54RSPABLG3SA7WCAHZ3NGTNAT7Q2KA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284344186 (321h35m43.087808767s) [noise=true , address=145.79.227.5:4221 , piece_id=QJUM5TUUJOIEFOASZGIZZFR7MZNR2D5XNAOTLIGZATAJTIAKM4CQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.284597364 (321h35m43.088061956s) [noise=true , address=145.79.227.17:4969 , piece_id=OTC3Z2ITY4XSDP6KV46CGGBCZWM4OIWNF2PIYWSDRDW7M2FPPZTA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.289948030 (321h35m43.09341261s) [noise=true , address=145.79.227.17:4971 , piece_id=ROAYJGPFDR5Z3FWORXBHNI5DBNWRTYUWKSLQCUGIHLUC34HPUSXA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.290567653 (321h35m43.094032245s) [noise=true , address=145.79.227.7:4294 , piece_id=S3CQ4GTGFPP7QNWDIE2KH7LV5T53TP7GJX73JKQHRJ7LNJY6FM5Q, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:05:55.294140755 (321h35m43.097605346s) [noise=true , address=145.79.227.15:4873 , piece_id=HSRJPGKW3XPR5WTWHPLDKFWHEHKOMZACGGLGIY4VJWZDLOJ6R6OQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: <nil>
Oct 6 23:06:24.065676552 (321h36m11.869141133s) [noise=true , address=45.91.116.13:4155 , piece_id=VKDKWKBJMGQU2RF2YGHZRURB6G5RUEE4EWI5QUOCOBGB55D5BVXQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 139.59.144.22:41710->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065700549 (321h36m11.869165133s) [noise=true , address=77.90.31.15:4699 , piece_id=NKNPV3ZNYTULY2LIAMUYVCOA6XFJEA4NKHHMLK2QJTKKZUJQTRGA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 139.59.144.22:37320->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.065859567 (321h36m11.869324164s) [noise=true , address=45.91.116.7:4009 , piece_id=TK5IBAG5X7VT4CKQJDEPFIG2CH3WCMU7ILIHHUIT4652GTN2TSZQ, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 139.59.144.22:33926->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066062619 (321h36m11.869527291s) [noise=true , address=45.91.116.12:4226 , piece_id=ZTT4DPSZKRNS337CZ6AGTIXGCNBXZWGCR62V2GCEGMTRB3ZI4FUA, seg_pos={0 0}, upload=283012]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: rpc: dial tcp 45.91.116.12:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:06:24.066099109 (321h36m11.869563688s) [seg_pos={0 0}, upload=283012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 6 23:06:24.180602946 (321h36m11.984067532s) [upload=283012]: Done uploading segments...
Oct 6 23:06:24.180634103 (321h36m11.984098694s) [upload=283012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.500867604s
core
uplink
us-select-1
|