Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................................X.............] duration:
1.634555462s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:38:32 UTC)
orbiter warning: context deadline exceeded
Duration: 1.634579769s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............................................X............] duration:
1.744037933s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:37:28 UTC)
orbiter warning: context deadline exceeded
Duration: 1.74411779s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................................X...........] duration:
1.60953514s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:36:26 UTC)
orbiter warning: context deadline exceeded
Duration: 1.609563831s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................................X..........] duration:
1.674016083s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:35:59 UTC)
orbiter warning: context deadline exceeded
Duration: 1.674053345s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........] duration:
1.676366603s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:35:08 UTC)
orbiter warning: context deadline exceeded
Duration: 1.676395799s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........] duration:
1.592127613s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:34:15 UTC)
orbiter warning: context deadline exceeded
Duration: 1.592157258s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......] duration:
1.674476154s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:33:32 UTC)
orbiter warning: context deadline exceeded
Duration: 1.674789609s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......] duration:
1.702119695s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:33:07 UTC)
orbiter warning: context deadline exceeded
Duration: 1.702152538s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....] duration:
1.806834635s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:32:04 UTC)
orbiter warning: context deadline exceeded
Duration: 1.806863796s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................................................X....] duration:
1.907630308s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:31:24 UTC)
orbiter warning: context deadline exceeded
Duration: 1.907715309s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
1.694015768s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:31:06 UTC)
orbiter warning: context deadline exceeded
Duration: 1.694059475s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
1.650788313s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:30:31 UTC)
orbiter warning: context deadline exceeded
Duration: 1.650822873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
1.813831669s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:30:12 UTC)
orbiter warning: context deadline exceeded
Duration: 1.813865807s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.000974817s trace:
https://jaeger.dev.storj.io/trace/7bc12ef83c7919a1
FAIL (2026-02-02 12:29:13 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001011872s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.388915039s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:11:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.388942576s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.435060905s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:10:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.435094782s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.375737224s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:10:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.376011029s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.246617073s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:09:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.246659488s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
2.376964013s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:08:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.377012429s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
2.328542835s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:07:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.328617753s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
2.392557351s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:07:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.392588118s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.309673038s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:06:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.309704288s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
2.308616339s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:06:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.308645789s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.365234818s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:05:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.365264264s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
2.304983687s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:04:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.305010337s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
2.327521141s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:03:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.327545776s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
2.310400078s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:02:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.310431756s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
2.261672498s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:02:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.261699563s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
2.318326564s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:01:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.318358269s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
2.309735786s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:01:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.309762827s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
2.736870623s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 23:00:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.73689791s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
2.313167345s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:59:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.313332517s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
2.280577087s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:59:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.280603563s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
2.252258964s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:58:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.252298073s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
2.38669187s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:57:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.386721627s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.35772617s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:56:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.357754909s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
2.299502605s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:56:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.299601992s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.309531953s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:55:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.30955881s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
2.309099674s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:54:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.30912547s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.354966654s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:54:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.354991115s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.236572246s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:53:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.236601042s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.312515148s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:51:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.31254706s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.315720597s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:50:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.315746316s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
2.399150727s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:50:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.399168007s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.329574741s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:48:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.329721817s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.338631098s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:47:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.338668197s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
2.266792384s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:47:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.266828559s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
2.253445374s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:47:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.253476017s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.2985757s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:46:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.298602839s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.298978384s trace:
https://jaeger.dev.storj.io/trace/30230c620b845632
FAIL (2026-02-01 22:45:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ) context deadline exceeded; piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ) context deadline exceeded; piecestore: (Node ID: 1u4KzGpsX41P8U5ZHLcjZMDGPEEF6R1CcNeTmp8NsPLkj9Putd, Piece ID: 7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ) context deadline exceeded
======LOGS======
rting upload
Feb 1 22:29:17.097065040 (414h33m54.981019243s) [upload=364705]: Done starting upload
Feb 1 22:29:17.097122569 (414h33m54.981076771s) [upload=364705]: Uploading segments...
Feb 1 22:29:17.097334052 (414h33m54.981288252s) [upload=364705]: Got next segment. Inline: false
Feb 1 22:29:17.097372902 (414h33m54.981327106s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 1 22:29:17.097382566 (414h33m54.981336768s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 1 22:29:17.335154673 (414h33m55.219108875s) [seg_pos={0 0}, upload=364705]: Begin upload segment...
Feb 1 22:29:17.335214777 (414h33m55.219168962s) [seg_pos={0 0}, upload=364705]: Done begin upload segment.
Feb 1 22:29:17.335243988 (414h33m55.21920992s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335270811 (414h33m55.219225014s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.335683574 (414h33m55.21963776s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.336329217 (414h33m55.220283425s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.337184254 (414h33m55.221138446s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338162334 (414h33m55.222116536s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338589715 (414h33m55.222543914s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.338970015 (414h33m55.222924214s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.339929314 (414h33m55.223883521s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.340432301 (414h33m55.224386506s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.341729997 (414h33m55.225684208s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.344593932 (414h33m55.228548136s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.345960875 (414h33m55.229915084s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.349088320 (414h33m55.233042522s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.351346093 (414h33m55.235300303s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:17.355249991 (414h33m55.239204196s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Uploading piece...
Feb 1 22:29:18.158784611 (414h33m56.042738802s) [upload=364705]: Next returned nil segment
Feb 1 22:29:18.158799506 (414h33m56.042753706s) [upload=364705]: Waiting for error group managing segments...
Feb 1 22:29:18.318055393 (414h33m56.202009592s) [noise=true , address=145.79.227.48:4303 , piece_id=BY43CNMHXCTFZOSSNULOO4TR27B4YE7VFYUFAKC34D6RT7ABSKHQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.326949426 (414h33m56.210903615s) [noise=true , address=145.79.227.47:4265 , piece_id=SELVSN6KME2KWEF7VGGB6TEV6BFCCEMZHFWD3Y7LR2SJR3LDVE2Q, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.338836804 (414h33m56.22279101s) [noise=true , address=45.91.116.4:4952 , piece_id=LR5IYPCVPAGLYUHVCDAGMGRGCDVIAP3DQOSKMGHGF3KKWAD3PKCA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.347130482 (414h33m56.231084682s) [noise=true , address=145.79.227.29:4943 , piece_id=HUMNAP5FRVP5PRZ4NZ52C3GSJGZ5AAKAUREXOA257YDZLXR4RNMA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.354621383 (414h33m56.238575588s) [noise=true , address=45.91.116.12:4228 , piece_id=6ORCZCBTMORCNDU4N4MM42J6Z2JPYBMTH3YGVJGGYGRIMPCHN5EQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.452677102 (414h33m56.336631302s) [noise=true , address=45.91.116.8:4070 , piece_id=BJRLDCNCTMHZFJ5A3JIPP6DTTKFUOV7C66YNTCRVU3ZVVYPCSPMQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.453904919 (414h33m56.337859118s) [noise=true , address=45.91.116.9:4263 , piece_id=QLD2WX72V3TI7UX2NKDFO2LZXFTVPC4IPYTF3KC6VI2TXEL5NSFA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:18.496669154 (414h33m56.380623354s) [noise=true , address=145.79.227.49:4337 , piece_id=TFZV2EW4BNRPUS3VRLYUD75JN7FHHGSBUCCE7TFCENFXJMIPJFDQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:25.780280557 (414h34m3.664234751s) [noise=true , address=45.91.116.7:4012 , piece_id=CWOB3Y4EOZFRMJFT55ZOQG4DROX3QTJMAY5M2YZPRZ4KHAF5FZTA, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:30.888263039 (414h34m8.772217242s) [noise=true , address=145.79.227.46:4601 , piece_id=VNCIW3HRVWTGVC6YJLLFI5ZDHCYIYPZ5XWOKZD2ISI3TV3MNQCJQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:31.096281792 (414h34m8.980235992s) [noise=true , address=45.91.116.7:4026 , piece_id=7OFVR6YRQOTDJI5FK2ARKGS4IB6E23SHN4VVSXQ6XCDR5P6FKETQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:38.023583467 (414h34m15.907537659s) [noise=true , address=145.79.227.35:4481 , piece_id=HPROZBOKKDSNP7APCDIWH3YEVL6TCYEVU4A7JDAUAJG5LKKTD4WQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502030941 (414h34m19.385985142s) [noise=true , address=45.91.116.10:4082 , piece_id=3RJRH5YKAPIXRPNAG2KTHG2FCQDY4UQ5MTZ5E7GGZEIUAQFSIKRQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: <nil>
Feb 1 22:29:41.502055111 (414h34m19.386009314s) [seg_pos={0 0}, upload=364705]: Satisfied cohort requirements after 13 pieces
Feb 1 22:29:41.502424056 (414h34m19.386378281s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: read tcp 139.59.144.22:46532->145.79.227.44:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502436485 (414h34m19.386390691s) [noise=true , address=145.79.227.44:4193 , piece_id=ABLJ62MDO3SSNYAOAFLXFTNZYVNXAQJFNT6YW26LNY7RI4OMFOUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502653332 (414h34m19.386607533s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 139.59.144.22:34080->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502665251 (414h34m19.38661945s) [noise=true , address=145.79.227.30:4762 , piece_id=XI6P33L4CSBAMZDDOYDTNYEZYT3GVJH6WCGYZWXTKOOUOJFH7BUQ, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502870835 (414h34m19.386825044s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: read tcp 139.59.144.22:32998->145.79.227.35:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 1 22:29:41.502881828 (414h34m19.386836032s) [noise=true , address=145.79.227.35:4482 , piece_id=A6RHMJWRT6VJVH267SIUUXAL4JV4QGAIJOV7GGH6MLJG65ESJV5A, seg_pos={0 0}, upload=364705]: Long tail cancellation due to optimal threshold
Feb 1 22:29:41.502905946 (414h34m19.386860146s) [seg_pos={0 0}, upload=364705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Feb 1 22:29:41.503017232 (414h34m19.386971438s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Feb 1 22:29:41.503025370 (414h34m19.386979579s) [upload=364705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Feb 1 22:29:41.658389046 (414h34m19.542343249s) [upload=364705]: Done uploading segments...
Feb 1 22:29:41.658410012 (414h34m19.542364212s) [upload=364705]: Upload finished. err: <nil>
Duration: 2.299008278s
core
uplink
us-select-1
|