Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X..........X......XX...X...............................] duration:
2.688482791s
FAIL (2025-10-06 23:40:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.688520704s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X................X..........................................] duration:
2.656336881s
FAIL (2025-10-06 23:40:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.65639206s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X.......X...X..X..X.................................] duration:
2.507926483s
FAIL (2025-10-06 23:40:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.507984541s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X..........X......XX...X..............................] duration:
2.970498268s
FAIL (2025-10-06 23:39:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.97053157s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.X.......X.......X..X...X.........X.........................] duration:
3.063702844s
FAIL (2025-10-06 23:39:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.063730493s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X................X.........................................] duration:
2.811041737s
FAIL (2025-10-06 23:39:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.811064709s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X.......X...X..X..X................................] duration:
2.585576504s
FAIL (2025-10-06 23:39:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.585669448s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X..........X......XX...X.............................] duration:
2.930333479s
FAIL (2025-10-06 23:39:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.930365555s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..X.......X.......X..X...X.........X........................] duration:
3.887118931s
FAIL (2025-10-06 23:38:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.887149091s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X................X........................................] duration:
2.670368793s
FAIL (2025-10-06 23:38:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.670390556s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [X.........X.......X...X..X..X...............................] duration:
2.531064012s
FAIL (2025-10-06 23:38:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.531164555s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X................X.......................................] duration:
2.649161277s
FAIL (2025-10-06 23:38:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.649184089s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...X.......X.......X..X...X.........X.......................] duration:
2.882969549s
FAIL (2025-10-06 23:38:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.882994551s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X..........X......XX...X............................] duration:
2.723384153s
FAIL (2025-10-06 23:37:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.723431389s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X.........X.......X...X..X..X..............................] duration:
2.722951005s
FAIL (2025-10-06 23:37:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.723024202s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X................X......................................] duration:
2.664504326s
FAIL (2025-10-06 23:37:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.664533855s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X........X..........X......XX...X...........................] duration:
2.548936704s
FAIL (2025-10-06 23:37:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.548967283s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [....X.......X.......X..X...X.........X......................] duration:
2.666013843s
FAIL (2025-10-06 23:37:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.666037056s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X.........X.......X...X..X..X.............................] duration:
2.480222372s
FAIL (2025-10-06 23:37:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.480251873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X........X..........X......XX...X..........................] duration:
2.909663515s
FAIL (2025-10-06 23:37:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.90969273s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X................X.....................................] duration:
2.672042446s
FAIL (2025-10-06 23:36:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.672063562s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.....X.......X.......X..X...X.........X.....................] duration:
2.689349495s
FAIL (2025-10-06 23:36:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.68938981s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X.........X.......X...X..X..X............................] duration:
2.43061332s
FAIL (2025-10-06 23:36:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.430641012s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X........X..........X......XX...X.........................] duration:
2.520885982s
FAIL (2025-10-06 23:35:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.520944597s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X................X....................................] duration:
2.681284502s
FAIL (2025-10-06 23:35:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.68129828s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X........X..........X......XX...X........................] duration:
2.492218411s
FAIL (2025-10-06 23:35:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.492258542s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [......X.......X.......X..X...X.........X....................] duration:
2.787069709s
FAIL (2025-10-06 23:35:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.78711716s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X.........X.......X...X..X..X...........................] duration:
2.629380107s
FAIL (2025-10-06 23:35:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.629433643s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X................X...................................] duration:
2.717001344s
FAIL (2025-10-06 23:34:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.717024439s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X................X..................................] duration:
2.80459476s
FAIL (2025-10-06 23:34:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.804624735s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.......X.......X.......X..X...X.........X...................] duration:
2.708350035s
FAIL (2025-10-06 23:34:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.708382969s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X........X..........X......XX...X.......................] duration:
2.994084004s
FAIL (2025-10-06 23:34:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.994109278s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X.........X.......X...X..X..X..........................] duration:
2.789571707s
FAIL (2025-10-06 23:34:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.789593605s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X........X..........X......XX...X......................] duration:
2.707420787s
FAIL (2025-10-06 23:33:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.707454446s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X................X.................................] duration:
2.745862932s
FAIL (2025-10-06 23:33:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.74610133s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X.........X.......X...X..X..X.........................] duration:
2.468698103s
FAIL (2025-10-06 23:33:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.46872894s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [........X.......X.......X..X...X.........X..................] duration:
2.843799394s
FAIL (2025-10-06 23:33:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.843824164s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......X.........X.......X...X..X..X........................] duration:
2.547930633s
FAIL (2025-10-06 23:33:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.5479632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........X.......X.......X..X...X.........X.................] duration:
2.900414151s
FAIL (2025-10-06 23:33:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.900435292s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [........X.........X.......X...X..X..X.......................] duration:
2.531779462s
FAIL (2025-10-06 23:33:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.531820436s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X........X..........X......XX...X.....................] duration:
2.835097178s
FAIL (2025-10-06 23:33:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.835125195s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X................X................................] duration:
2.62048059s
FAIL (2025-10-06 23:33:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.620509438s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..........X.......X.......X..X...X.........X................] duration:
3.384543419s
FAIL (2025-10-06 23:32:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.384573278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......X........X..........X......XX...X....................] duration:
2.558491307s
FAIL (2025-10-06 23:32:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.558619444s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........X.........X.......X...X..X..X......................] duration:
2.579863031s
FAIL (2025-10-06 23:32:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.579889754s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X................X...............................] duration:
2.954089676s
FAIL (2025-10-06 23:32:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.954224453s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........X........X..........X......XX...X...................] duration:
2.709144789s
FAIL (2025-10-06 23:31:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
21h44m24.814944914s) [upload=283130]: Uploading segments...
Oct 6 23:14:37.011693674 (321h44m24.815158266s) [upload=283130]: Got next segment. Inline: false
Oct 6 23:14:37.011759368 (321h44m24.815223953s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:37.011765154 (321h44m24.815229745s) [upload=283130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:37.123040562 (321h44m24.926505157s) [seg_pos={0 0}, upload=283130]: Begin upload segment...
Oct 6 23:14:37.123145277 (321h44m24.926609876s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.130217212 (321h44m24.933681813s) [seg_pos={0 0}, upload=283130]: Done begin upload segment.
Oct 6 23:14:37.130354898 (321h44m24.9338195s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.131128462 (321h44m24.934593068s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.132696720 (321h44m24.93616132s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133596422 (321h44m24.937061023s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.133713183 (321h44m24.937177794s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134356842 (321h44m24.937821441s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134498352 (321h44m24.937962958s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.134905544 (321h44m24.938370149s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135151391 (321h44m24.938615995s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135465599 (321h44m24.938930197s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.135773195 (321h44m24.939237796s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136003505 (321h44m24.939468108s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.136665066 (321h44m24.940129665s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137195600 (321h44m24.940660202s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.137728259 (321h44m24.94119286s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Uploading piece...
Oct 6 23:14:37.785112144 (321h44m25.588576747s) [upload=283130]: Next returned nil segment
Oct 6 23:14:37.785138139 (321h44m25.58860274s) [upload=283130]: Waiting for error group managing segments...
Oct 6 23:14:37.912422203 (321h44m25.715886795s) [noise=true , address=145.79.227.18:4007 , piece_id=2QZNXC7KM3XJVZEEXSFOAFV2BLKBH57HE3B6KJ6DH2K5TQ7DNJCQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913097624 (321h44m25.716562216s) [noise=true , address=145.79.227.2:4115 , piece_id=FBP7E4QCAAMRKV46VDQAYCYZ3FVGVFL7KSU54X4NNOYBAYM6RZXA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913185687 (321h44m25.716650272s) [noise=true , address=145.79.227.19:4078 , piece_id=FAEFPSXXUIQE3CI5HTFKIVLUECVWYGKMIRZZYFTU4CXXUFDQ3K7Q, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913592252 (321h44m25.71705685s) [noise=true , address=145.79.227.18:4006 , piece_id=2ZH7VYG2Y2HCLIIT6UVS6UDAAWVWT3HGVGNZ2PEQI2JGQ3VJPBMA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.913974583 (321h44m25.717439169s) [noise=true , address=145.79.227.5:4223 , piece_id=MVLUJQ7ONSZIQ7TJQ5WWKKH5663CUMYRPTEXWY7T6TVZQ7ABPJGA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.916344832 (321h44m25.719809414s) [noise=true , address=145.79.227.3:4151 , piece_id=IKJPGOVYDETCQZEKVAQ72OROPK2W3MKNYEYXR2R7WRMAXQQ2QHQQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.921531945 (321h44m25.724996523s) [noise=true , address=145.79.227.15:4874 , piece_id=WCPDPEHBTWAIJIKLEVMN25KVEEDIQAQI75NBTQFSYZKXB4U4TQRQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:14:37.932105629 (321h44m25.735570208s) [noise=true , address=145.79.227.3:4150 , piece_id=HZ6KN5IV56KAOUGNSAKZFBST6A7PZAZC7CLALD2BF6OWUM72BVBA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: <nil>
Oct 6 23:15:06.715286602 (321h44m54.518751199s) [noise=true , address=45.91.116.11:4117 , piece_id=AO5N4ME4S3VAJ2SACTO3H47CT52C5FYRBXOOFSAVUX3OQNE7QSSA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: rpc: dial tcp 45.91.116.11:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715572308 (321h44m54.519036908s) [noise=true , address=45.91.116.4:4940 , piece_id=LAHVXBQ3UYSKIUCFRDVSOCJRR4FRS7ZDTSH5LUJHBTTOOH6X7KKQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715766153 (321h44m54.519230745s) [noise=true , address=45.91.116.4:4937 , piece_id=IY3XJTOOIQKKJX2Q6KLAFQEN3K4BGGZAJLP6TLLC5XWAVCN4EZAA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: rpc: dial tcp 45.91.116.4:4937: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.715993973 (321h44m54.519458565s) [noise=true , address=45.91.116.9:4262 , piece_id=P5VZRXD7JTE32FHSR2WZPQVDHGAQYLPKYJER43MTAWU5TZFYFIRA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 139.59.144.22:52202->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716157018 (321h44m54.519621613s) [noise=true , address=45.91.116.8:4045 , piece_id=PGZD2XHSBC3QTAKL7HB2KGVA43UKLWXOURLHYBRIEDIDKKK2AGVA, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:56836->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716311274 (321h44m54.519775865s) [noise=true , address=77.90.31.12:4590 , piece_id=LV635WNGGSTIAYP6CW5T6756ZS5ZRL5HUPFTH3EVKH3JE7VB4UNQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 139.59.144.22:60484->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716753675 (321h44m54.520218267s) [noise=true , address=77.90.31.12:4592 , piece_id=VIHVVY3HJXCUZWI2U3ZXN2JGLZFXK6MB5NZJKLVRG4WE5YP2RVDQ, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 139.59.144.22:59820->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716937387 (321h44m54.520401984s) [noise=true , address=45.91.116.5:4298 , piece_id=DDM3LFWCP3XJDDILAPCRGRG25XEID2VXHGCN2NJOBPYQCV2GFR5A, seg_pos={0 0}, upload=283130]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 139.59.144.22:52132->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:06.716972021 (321h44m54.520436618s) [seg_pos={0 0}, upload=283130]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 6 23:15:06.823750501 (321h44m54.627215092s) [upload=283130]: Done uploading segments...
Oct 6 23:15:06.823805470 (321h44m54.627270074s) [upload=283130]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.709175009s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...........X.......X.......X..X...X.........X...............] duration:
3.004547432s
FAIL (2025-10-06 23:31:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.004576146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........X.........X.......X...X..X..X.....................] duration:
2.482018538s
FAIL (2025-10-06 23:31:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
:14:40.114564120 (321h44m27.918028721s) [upload=283133]: Got next segment. Inline: false
Oct 6 23:14:40.114637926 (321h44m27.918102522s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:14:40.114645588 (321h44m27.918110194s) [upload=283133]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:40.234782987 (321h44m28.038247566s) [seg_pos={0 0}, upload=283133]: Begin upload segment...
Oct 6 23:14:40.234869686 (321h44m28.038334277s) [seg_pos={0 0}, upload=283133]: Done begin upload segment.
Oct 6 23:14:40.234988877 (321h44m28.038453472s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235049392 (321h44m28.038513986s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.235610571 (321h44m28.039075214s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.236286136 (321h44m28.039750739s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237036602 (321h44m28.040501193s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.237484234 (321h44m28.040948826s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238009813 (321h44m28.041474414s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.238662655 (321h44m28.042127268s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239160195 (321h44m28.042624779s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239295200 (321h44m28.04275978s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.239764845 (321h44m28.043229424s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240641257 (321h44m28.044105848s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.240760607 (321h44m28.044225207s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241175381 (321h44m28.044639975s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241616623 (321h44m28.045081215s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.241857613 (321h44m28.045322217s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242216650 (321h44m28.045681244s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.242561429 (321h44m28.04602602s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Uploading piece...
Oct 6 23:14:40.817451359 (321h44m28.620915951s) [upload=283133]: Next returned nil segment
Oct 6 23:14:40.817469197 (321h44m28.620933791s) [upload=283133]: Waiting for error group managing segments...
Oct 6 23:14:40.964356063 (321h44m28.767820647s) [noise=true , address=145.79.227.21:4501 , piece_id=GEYOLDIMYZBK4GJ5PNKOQTQLXAXKKOOFOAFFA3UV6HQ77L4TT63A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.967615781 (321h44m28.771080358s) [noise=true , address=145.79.227.32:4834 , piece_id=3JNT3GX5EMGON7RQJKWUXRDDNSHWU35YGHIARPVFFQ4J4J2IHHOA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971520114 (321h44m28.774984697s) [noise=true , address=145.79.227.16:4940 , piece_id=WWWR2LVOH7B2SZAXYMAW7JF4KLAKCJTHAPOVDOD37QJ22SQ4Y3MA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.971634950 (321h44m28.775099534s) [noise=true , address=145.79.227.13:4806 , piece_id=623JR4BIGPX22WONG57VN6U7V7ZLLAG4EIJATDZ4HQDDCZTWZKPA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.972075821 (321h44m28.775540404s) [noise=true , address=145.79.227.41:4094 , piece_id=W62USHBCFXW5XMDKS42J4NUAOZSSM7EVA25MT54UPS4LJUEZRY3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.994068275 (321h44m28.797532859s) [noise=true , address=145.79.227.27:4714 , piece_id=MZA424ADSRXBIBMEPPCWHJPDLRIJIUBV23NZ22HS3K43RWTO35YQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:40.996802013 (321h44m28.800266591s) [noise=true , address=145.79.227.28:4743 , piece_id=6WQFTXPRGMFV3Y5HYAGOKKAFWGNCNCLD43QXVRFC37J3AASMJ7BA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.039669025 (321h44m28.843133624s) [noise=true , address=145.79.227.33:4895 , piece_id=JCKD4RRUZLSB47I4WULRQKH22R42AEMZKUTSKYSSWZXMY7KPMPBQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.549371356 (321h44m29.35283595s) [noise=true , address=64.49.13.26:4956 , piece_id=TY4ZGSVIDE576GMJD374UG2B2HAJ4JJ3QAGH4IDZOXAFO3YOC37A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:14:41.556112439 (321h44m29.359577038s) [noise=true , address=64.49.13.27:4967 , piece_id=6Q4LKMKTS754CDFRVRPE2G7G2WRMONESTQKEFZ7FUTUIPXPVT6ZQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.486669475 (321h45m21.290134069s) [noise=true , address=77.90.31.17:4775 , piece_id=CSMWR3GLQV4ZJZR52K4A6ORLKSGJK47QTOD6H3XOCXJKS45QIW5A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:33.525025992 (321h45m21.32849057s) [noise=true , address=45.91.116.8:4060 , piece_id=45NSYIADUNFCZEK23PEKRJB5AATQAJ5AG24SFVB5THRMZVOPPTFQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:35.522001875 (321h45m23.325466454s) [noise=true , address=77.90.31.10:4522 , piece_id=K4FBSO53SABCWGRYRY4FCGQPSLHNIO6IWEFOCFKPS772UORNWEDA, seg_pos={0 0}, upload=283133]: Done uploading piece. err: <nil>
Oct 6 23:15:39.810917110 (321h45m27.61438169s) [noise=true , address=77.90.31.18:4835 , piece_id=ZEIA6RG2GCCFVWEEQWGUFWQGY2UAZ5GPJVBRQOLNN3WJXVUJEZ2A, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.810972535 (321h45m27.614437116s) [noise=true , address=77.90.31.11:4584 , piece_id=FII3T3RXJLSSBHQI3UC4PUF3EELRPT5ZAUQUG2DRACM76G5AYHGQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811092364 (321h45m27.614556942s) [noise=true , address=45.91.116.7:4021 , piece_id=BTKLX4QVZEAXFR2H3WNJVUFNOZY7HEG7PCWNF5L3KECL2HDUXU3Q, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1bXLNG7g5RJc21AjrfHup6LDTd2dDJGWMzrBhVYwUT7fNm8J2Z, address:45.91.116.7:4021): protocol: read tcp 139.59.144.22:51136->45.91.116.7:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811128811 (321h45m27.614593393s) [noise=true , address=77.90.31.12:4617 , piece_id=7QGFJDCAUPQ3OQY3J32A5RPGBXNKQYPSGNSKWEZ2CP6YPZSJNFSQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: read tcp 139.59.144.22:52612->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:39.811267995 (321h45m27.614732571s) [noise=true , address=77.90.31.13:4642 , piece_id=K5SJ474XLG4RKZ5OVDELV5EJWCLKE7P7QJDXLQFFF5MY5TE2VEDQ, seg_pos={0 0}, upload=283133]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: write tcp 139.59.144.22:34550->77.90.31.13:4642: use of closed network connection; conn closed: op
Oct 6 23:15:39.811289689 (321h45m27.614754266s) [seg_pos={0 0}, upload=283133]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Oct 6 23:15:39.920043283 (321h45m27.723507868s) [upload=283133]: Done uploading segments...
Oct 6 23:15:39.920082671 (321h45m27.723547251s) [upload=283133]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.482046005s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X................X..............................] duration:
2.553251165s
FAIL (2025-10-06 23:31:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.55328598s
edge
gateway
us-select-1
|