Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........X........X..........X......XX...X..................] duration:
2.689875005s
FAIL (2025-10-06 23:31:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.68991338s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........X.........X.......X...X..X..X....................] duration:
2.462789231s
FAIL (2025-10-06 23:30:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.462822539s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X................X.............................] duration:
2.795918791s
FAIL (2025-10-06 23:30:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.795939496s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [............X.......X.......X..X...X.........X..............] duration:
2.807470816s
FAIL (2025-10-06 23:30:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.807502378s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........X........X..........X......XX...X.................] duration:
2.667514244s
FAIL (2025-10-06 23:30:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.667548911s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [............X.........X.......X...X..X..X...................] duration:
2.670290978s
FAIL (2025-10-06 23:29:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.670316795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.............X.......X.......X..X...X.........X.............] duration:
2.924243297s
FAIL (2025-10-06 23:29:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.924267136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X................X............................] duration:
2.668800296s
FAIL (2025-10-06 23:29:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.66890462s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............X.........X.......X...X..X..X..................] duration:
2.374769128s
FAIL (2025-10-06 23:29:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.374947775s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........X........X..........X......XX...X................] duration:
2.719484877s
FAIL (2025-10-06 23:29:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.719547669s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X................X...........................] duration:
2.833213569s
FAIL (2025-10-06 23:28:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.833246051s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............X.........X.......X...X..X..X.................] duration:
2.41966628s
FAIL (2025-10-06 23:28:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.419697504s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..............X.......X.......X..X...X.........X............] duration:
2.944416247s
FAIL (2025-10-06 23:28:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.944437148s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............X........X..........X......XX...X...............] duration:
2.5362271s
FAIL (2025-10-06 23:28:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.536257246s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............X.........X.......X...X..X..X................] duration:
2.527965261s
FAIL (2025-10-06 23:27:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.527994975s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X................X..........................] duration:
2.85939054s
FAIL (2025-10-06 23:27:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.860283551s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [................X.........X.......X...X..X..X...............] duration:
2.403581367s
FAIL (2025-10-06 23:27:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.403610196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...............X.......X.......X..X...X.........X...........] duration:
3.363738842s
FAIL (2025-10-06 23:27:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.36376385s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............X........X..........X......XX...X..............] duration:
2.503500202s
FAIL (2025-10-06 23:27:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.503530184s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X................X.........................] duration:
2.842770088s
FAIL (2025-10-06 23:27:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.842803159s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [................X.......X.......X..X...X.........X..........] duration:
2.964543068s
FAIL (2025-10-06 23:26:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.964765647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................X.........X.......X...X..X..X..............] duration:
2.487836796s
FAIL (2025-10-06 23:26:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.487863268s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............X........X..........X......XX...X.............] duration:
2.587320043s
FAIL (2025-10-06 23:26:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.587352414s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.................X.......X.......X..X...X.........X.........] duration:
3.386280713s
FAIL (2025-10-06 23:26:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.38631138s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X................X........................] duration:
2.677541629s
FAIL (2025-10-06 23:26:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.677576779s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..................X.........X.......X...X..X..X.............] duration:
2.638104621s
FAIL (2025-10-06 23:26:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.638140559s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..................X.......X.......X..X...X.........X........] duration:
2.822184704s
FAIL (2025-10-06 23:25:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.822621968s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X................X.......................] duration:
2.532066762s
FAIL (2025-10-06 23:25:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.532128236s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................X.........X.......X...X..X..X............] duration:
2.38103137s
FAIL (2025-10-06 23:25:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.381160378s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............X........X..........X......XX...X............] duration:
2.759876244s
FAIL (2025-10-06 23:25:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.759912999s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...................X.......X.......X..X...X.........X.......] duration:
2.888140202s
FAIL (2025-10-06 23:24:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.888164254s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [....................X.........X.......X...X..X..X...........] duration:
2.573312835s
FAIL (2025-10-06 23:24:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.57333519s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X................X......................] duration:
2.772318852s
FAIL (2025-10-06 23:24:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.772878637s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [................X........X..........X......XX...X...........] duration:
2.744171131s
FAIL (2025-10-06 23:24:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.744201684s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [....................X.......X.......X..X...X.........X......] duration:
2.924881712s
FAIL (2025-10-06 23:24:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.924907549s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X................X.....................] duration:
2.845893235s
FAIL (2025-10-06 23:23:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.845917392s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....................X.........X.......X...X..X..X..........] duration:
2.411423173s
FAIL (2025-10-06 23:23:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.411526821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.....................X.......X.......X..X...X.........X.....] duration:
2.801151107s
FAIL (2025-10-06 23:23:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.801177963s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................X........X..........X......XX...X..........] duration:
2.630768937s
FAIL (2025-10-06 23:23:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.63080115s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................X.........X.......X...X..X..X.........] duration:
2.640173253s
FAIL (2025-10-06 23:22:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.640200955s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X................X....................] duration:
2.632508855s
FAIL (2025-10-06 23:22:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.632537548s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [......................X.......X.......X..X...X.........X....] duration:
2.972154834s
FAIL (2025-10-06 23:22:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.972192814s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..................X........X..........X......XX...X.........] duration:
2.701628179s
FAIL (2025-10-06 23:22:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.701660663s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X................X...................] duration:
2.676556589s
FAIL (2025-10-06 23:21:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.676580712s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................X........X..........X......XX...X........] duration:
2.778755752s
FAIL (2025-10-06 23:21:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.778786563s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......................X.........X.......X...X..X..X........] duration:
2.518698563s
FAIL (2025-10-06 23:21:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.518736995s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.......................X.......X.......X..X...X.........X...] duration:
3.48869146s
FAIL (2025-10-06 23:21:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.488720201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [........................X.......X.......X..X...X.........X..] duration:
3.029204671s
FAIL (2025-10-06 23:21:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.029230445s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X................X..................] duration:
2.524628231s
FAIL (2025-10-06 23:21:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.524654604s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [........................X.........X.......X...X..X..X.......] duration:
2.787669396s
FAIL (2025-10-06 23:21:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.78770494s
core
uplink
us-select-1
us1-placement-20
|