Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.978673206s trace:
https://jaeger.dev.storj.io/trace/3ef81919e1e402f0
FAIL (2026-02-10 12:40:32 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 28)
Duration: 2.978701158s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................................................XX........] duration:
1.782708281s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:40:15 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.782744518s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.927937013s trace:
https://jaeger.dev.storj.io/trace/3ef81919e1e402f0
FAIL (2026-02-10 12:39:42 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 28)
Duration: 2.927982165s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................................................XX.......] duration:
1.668086724s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:39:27 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.668117306s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................................................XX......] duration:
1.767146598s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:39:05 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.767255126s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
3.007758596s trace:
https://jaeger.dev.storj.io/trace/3ef81919e1e402f0
FAIL (2026-02-10 12:38:53 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 28)
Duration: 3.007809897s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration:
1.719414081s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:38:48 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.719441665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
3.091485704s trace:
https://jaeger.dev.storj.io/trace/3ef81919e1e402f0
FAIL (2026-02-10 12:37:58 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 28)
Duration: 3.091576283s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................................................XX....] duration:
1.740366968s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:37:58 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.74038898s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................................XX...] duration:
1.699255495s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:37:42 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.699288026s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................................XX..] duration:
1.657041669s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:37:07 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.657070821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
3.473171064s trace:
https://jaeger.dev.storj.io/trace/3ef81919e1e402f0
FAIL (2026-02-10 12:36:52 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 28)
Duration: 3.473202694s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................................XX.] duration:
1.697999799s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:36:35 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.698017287s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
31.942391917s trace:
https://jaeger.dev.storj.io/trace/3ef81919e1e402f0
FAIL (2026-02-10 12:36:26 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 28)
Duration: 31.94242225s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................................................XX] duration:
5.000678468s trace:
https://jaeger.dev.storj.io/trace/1d8b7df52c1b5a06
FAIL (2026-02-10 12:36:21 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000702474s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.001688591s trace:
https://jaeger.dev.storj.io/trace/188846c31963871a
FAIL (2026-02-10 12:35:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001709944s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
1.65284382s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:40:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.653088682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
1.630507205s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:39:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.630536565s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.457697405s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:39:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.457719727s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X.X.........................................................] duration:
1.622726848s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:39:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.622754788s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.279443645s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:39:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.279473559s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.X.X........................................................] duration:
1.691695012s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:38:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.691722604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.354737471s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:38:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.354763549s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.309353613s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:37:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.30938612s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..X.X.......................................................] duration:
1.598625646s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:37:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.598714477s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
2.419460522s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:37:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.419572262s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...X.X......................................................] duration:
2.13860682s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:36:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.138629904s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
2.336531471s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:36:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.33655697s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....X.X.....................................................] duration:
1.705849306s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:35:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.705873054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
2.365779988s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:35:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.365807836s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.364551616s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:35:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.364580381s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....X.X....................................................] duration:
1.668199438s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:34:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.668242986s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
2.296111869s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:34:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.296140089s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.326362575s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:34:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.326427369s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......X.X...................................................] duration:
1.725071995s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:34:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.725091007s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......X.X..................................................] duration:
1.628063406s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:33:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.628092307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
2.288476176s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:33:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.288522162s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........X.X.................................................] duration:
1.681350312s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:32:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.681374947s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
2.310981034s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:32:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.311013791s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........X.X................................................] duration:
1.718589477s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:31:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.718618855s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
2.274532128s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:30:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.274565181s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........X.X...............................................] duration:
1.560145533s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:30:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.560240151s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
2.507651894s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:30:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.507685035s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
2.272707569s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:30:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.272757305s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........X.X..............................................] duration:
1.638140691s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:29:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.638218005s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............X.X.............................................] duration:
1.775820216s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:29:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.775844074s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
2.385351629s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:28:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.38537958s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
2.345926552s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:28:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.345995194s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............X.X............................................] duration:
1.576153611s trace:
https://jaeger.dev.storj.io/trace/6390dd2a774c1a76
FAIL (2026-02-10 05:28:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.576183263s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
2.38483877s trace:
https://jaeger.dev.storj.io/trace/5c513512a856fd10
FAIL (2026-02-10 05:27:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Feb 10 04:56:26.407884743 (613h1m4.291838946s) [upload=539706]: Starting upload
Feb 10 04:56:26.407962948 (613h1m4.291917147s) [upload=539706]: Done starting upload
Feb 10 04:56:26.408281730 (613h1m4.292235929s) [upload=539706]: Uploading segments...
Feb 10 04:56:26.408355658 (613h1m4.292309857s) [upload=539706]: Got next segment. Inline: false
Feb 10 04:56:26.408391309 (613h1m4.29234551s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 10 04:56:26.408397227 (613h1m4.292351426s) [upload=539706]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 10 04:56:26.517830280 (613h1m4.401784487s) [seg_pos={0 0}, upload=539706]: Begin upload segment...
Feb 10 04:56:26.517986937 (613h1m4.401941137s) [seg_pos={0 0}, upload=539706]: Done begin upload segment.
Feb 10 04:56:26.518051397 (613h1m4.402005604s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.518683750 (613h1m4.402637959s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.519522687 (613h1m4.40347689s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520104774 (613h1m4.404058987s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520474071 (613h1m4.404428274s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520591983 (613h1m4.404546193s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.520889788 (613h1m4.404843987s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521201235 (613h1m4.405155444s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521363432 (613h1m4.405317639s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521678719 (613h1m4.405632953s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.521824152 (613h1m4.405778362s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.522382988 (613h1m4.406337191s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523006609 (613h1m4.406960819s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523530743 (613h1m4.407484953s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.523971213 (613h1m4.407925419s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:26.524555582 (613h1m4.408509767s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Uploading piece...
Feb 10 04:56:27.125437351 (613h1m5.009391551s) [upload=539706]: Next returned nil segment
Feb 10 04:56:27.125450506 (613h1m5.009404705s) [upload=539706]: Waiting for error group managing segments...
Feb 10 04:56:27.265993299 (613h1m5.149947498s) [noise=true , address=145.79.227.29:4944 , piece_id=EBZDHA5GT2MNJIHBMRZ4AUCYO2C4FUSJYRH4EOAT57HUP6D7WPMA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.267144887 (613h1m5.151099086s) [noise=true , address=145.79.227.10:4406 , piece_id=JDJYLQKJHME5YUW3CBV2W7K55ZGWAIMV3DVYEDSMGWZUR2LCC22Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268411655 (613h1m5.152365846s) [noise=true , address=145.79.227.35:4487 , piece_id=GFW5XK6HDN3RPUFGZJR3GEFUGLYNZ2ZZIPNKSZZQ7QUKR6X6LW6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.268424437 (613h1m5.152378629s) [noise=true , address=145.79.227.24:4587 , piece_id=667GJB73DZEQQL7PPU2QFMV5F4PBO7T27CPQJT64VZOIGXTZHK6A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269455913 (613h1m5.153410113s) [noise=true , address=145.79.227.18:4010 , piece_id=TESZWTBZVOBPUVKONQ2UKXFHR6UCZCG5HODGQFRBPSGQ5FXXZSPA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.269891228 (613h1m5.153845428s) [noise=true , address=145.79.227.12:4771 , piece_id=PH55TD2FHH4OSOANKY4GYUFSJTPY6GESHT466MK6MNRKDJ7ULN7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.276930861 (613h1m5.16088507s) [noise=true , address=145.79.227.26:4659 , piece_id=JHPHXRSV4TPKSJKU4OTGHXQJPAQUQJYLAKSYHSL5X4JCBPNNW7DQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.283467915 (613h1m5.167422119s) [noise=true , address=145.79.227.11:4735 , piece_id=EKNFOD5CXZPORL2SD7LNNIGL6EHJLHSQCM7ISHVQIVX3LG7NXEDA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.300848770 (613h1m5.184802976s) [noise=true , address=145.79.227.52:4451 , piece_id=S7FYVRYGC7HJMBNK677OJX7UJMUQ5WNMVNJ2NLUGPINA4M56VR2Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:27.355574655 (613h1m5.239528859s) [noise=true , address=145.79.227.44:4198 , piece_id=63QX7GNACZGMRL7EADJ6IULDWHQVFQ6AFXCDQTSUEEPRBRNZFWWA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.017008104 (613h1m5.900962305s) [noise=true , address=145.79.227.6:4260 , piece_id=IQZBRBB7IOSM5WHIGCAGGJS53AFDPEQ6CV7U7FSJG62LT5HFOX7Q, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:28.766902450 (613h1m6.650856651s) [noise=true , address=79.127.219.38:4817 , piece_id=LGBUXYBCCNMQKB6KBS6K6RTUB6MKKBFGITSAO7S7MRSGJHKEAGUA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: <nil>
Feb 10 04:56:56.130179108 (613h1m34.014133315s) [noise=true , address=77.90.31.12:4589 , piece_id=X63HQQ66ADLBBELZPBSUWLWRLDYIG3AAFMEPVUGPRGBVMC3THDZA, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60418->77.90.31.12:4589: use of closed network connection
Feb 10 04:56:56.130215419 (613h1m34.014169624s) [noise=true , address=77.90.31.19:4844 , piece_id=OXWT333ZD33B3S6UGCWPHR62PLG5R2KDSOXLF6W5CBWBBBFTEH2A, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: read tcp 139.59.144.22:57156->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130422685 (613h1m34.01437689s) [noise=true , address=77.90.31.13:4629 , piece_id=2WMG7SACVLNRXLVQLCM2QRXRHT3LV4VMK5YWLDQO5F5FBDT4PQNQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: read tcp 139.59.144.22:35224->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130571357 (613h1m34.014525563s) [noise=true , address=77.90.31.11:4553 , piece_id=VNIM353YMYAT2ZHXUSJZJRBFWG4NN7VMVKP37JK6KNJAIIJ5UKGQ, seg_pos={0 0}, upload=539706]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: read tcp 139.59.144.22:41684->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 10 04:56:56.130599829 (613h1m34.014554014s) [seg_pos={0 0}, upload=539706]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Feb 10 04:56:56.233108051 (613h1m34.117062253s) [upload=539706]: Done uploading segments...
Feb 10 04:56:56.233140386 (613h1m34.117094584s) [upload=539706]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.384952483s
core
uplink
us-select-1
|